AIML
Dr. Nitin A. Shelke
Dimensionality Reduction Techniques
• PCA
• Kernel PCA
• LDA
Need of Dimensionality Reduction
• Machine Learning in general works wonders when the dataset provided for
training the machine is large and concise.
• However, using a large data set has its own pitfalls. The biggest pitfall is the
curse of dimensionality.
• It turns out that in large dimensional datasets, there might be lots of
inconsistencies in the features or lots of redundant features in the dataset,
which will only increase the computation time and make data processing
convoluted.
• To get rid of the curse of dimensionality, a process called dimensionality
reduction was introduced.
PCA
• Principal Component Analysis (PCA) is a dimensionality reduction
technique or algorithm. PCA is defined by a transformation of a high
dimensional vector space into a low dimensional space.
• Moreover, PCA is an unsupervised statistical technique used to examine the
interrelations among a set of variables.
• The main idea behind PCA is to identify the patterns in the data and
correlations among various features in the data set. On finding a strong
correlation between different variables, a final decision is made about
reducing the dimensions of the data in such a way that the significant data
is still retained.
Example
Step By Step Computation Of PCA
• The below steps need to be followed to perform dimensionality reduction
using PCA:
• Standardization of the data
• Computing the covariance matrix
• Calculating the eigenvalues and eigenvectors
• Computing the Principal Components (New set of variables)
• Reducing the dimensions of the data set.
Note: PCA tries to compress as much information as possible in the first PC, the rest in
the second, and so on…
Example PCA
Example PCA
Example PCA
Example PCA
Example PCA
divide the eigenvalue of each component by the sum of
eigenvalues.
Kernel PCA
• PCA is a linear method. That is it can only be applied to datasets which
are linearly separable.
• It does an excellent job for datasets, which are linearly separable. But, if
we use it to non-linear datasets, we might get a result which may not be
the optimal dimensionality reduction.
• Kernel PCA uses a kernel function to project dataset into a higher
dimensional feature space, where it is linearly separable. It is similar to
the idea of Support Vector Machines.
Linear Discriminant Analysis (LDA)
• Both PCA and LDA are linear transformation techniques used for dimensionality reduction.
• However, PCA is an unsupervised while LDA is a supervised dimensionality reduction technique.
• PCA has no concern with the class labels. In simple words, PCA summarizes the feature set without relying
on the output.
• Unlike PCA, LDA tries to reduce dimensions of the feature set while retaining the information that
discriminates output classes.
• It is used for modeling differences in groups i.e. separating two or more classes.
• It is used to project the features in higher dimension space into a lower dimension space.
