Clustering

Clustering is an unsupervised learning technique used to group similar data points into clusters based on their
features.

1. Mixture Densities

 Definition: A statistical model that represents a distribution as a combination of multiple component

distributions, each associated with a different cluster.
 Example: A Gaussian mixture model (GMM) where data points can belong to multiple clusters with
different probabilities.

2. k-Means Clustering

 Definition: A popular clustering algorithm that partitions data into k clusters by minimizing the variance
within each cluster.
 Algorithm Steps:
1. Initialization: Choose k initial centroids randomly.
2. Assignment: Assign each data point to the nearest centroid.
3. Update: Recalculate the centroids as the mean of all points in the cluster.
4. Repeat: Continue until the assignments no longer change or a maximum number of iterations is
reached.
 Advantages: Simple, fast, and easy to implement.
 Disadvantages: Requires pre-specification of k, sensitive to initial centroid positions, and can converge
to local minima.

3. Expectation-Maximization (EM)

 Definition: A statistical technique for finding maximum likelihood estimates in models with latent
variables.
 Process:
1. Expectation Step (E-step): Estimate the expected value of the log-likelihood function, given the
current parameters.
2. Maximization Step (M-step): Maximize this expectation to update the parameters.
 Application: Often used with Gaussian mixture models to cluster data where clusters have different
shapes and orientations.

4. Mixtures of Latent Variable Models

 Definition: Models that assume the data is generated from a mixture of several latent (unobserved)
variables.
 Use Case: Useful for modeling complex data distributions and can represent hierarchies or interactions
between observed variables.

5. Supervised Learning after Clustering

 Definition: Applying supervised learning techniques to the clustered data to enhance prediction models.
 Process: After clustering, labels can be assigned to clusters, and these labels can be used as features in
supervised learning algorithms, improving model performance.
6. Hierarchical Clustering

 Definition: A clustering method that builds a hierarchy of clusters, either agglomeratively (bottom-up)
or divisively (top-down).
 Types:
o Agglomerative: Starts with each data point as its own cluster and merges them based on
distance.
o Divisive: Starts with one cluster and recursively splits it into smaller clusters.
 Dendrogram: A tree-like diagram that shows the arrangement of clusters based on distance or
similarity.

Decision Trees

Decision trees are a supervised learning technique used for classification and regression tasks. They model
decisions and their possible consequences in a tree-like structure.

1. Univariate Trees

 Definition: Trees that make decisions based on a single feature at each node.
 Construction: Each node represents a feature, and branches represent the decision based on that feature.

2. Classification Tree

 Purpose: Used for classifying data into distinct categories.

 Process:
1. Choose the best feature to split the data based on a criterion (e.g., Gini impurity, information
gain).
2. Repeat recursively until stopping criteria are met (e.g., maximum depth, minimum samples per
leaf).

3. Regression Trees

 Purpose: Used for predicting continuous outcomes.

 Structure: Similar to classification trees but splits the data to minimize the variance in the target
variable.

4. Pruning

 Definition: The process of removing sections of the tree that provide little predictive power to avoid
overfitting.
 Methods:
o Cost Complexity Pruning: Balances the tree's size and its accuracy.
o Reduced Error Pruning: Evaluates the effect of removing branches based on validation data.
5. Rule Extraction from Trees

 Definition: Deriving human-readable rules from decision trees.

 Process: Each path from the root to a leaf can be expressed as a rule that describes the conditions under
which a particular class is predicted.

6. Learning Rules from Data

 Definition: Extracting general rules or patterns from data, often using algorithms like Apriori or FP-
Growth for association rule mining.

7. Multivariate Trees

 Definition: Decision trees that can split on multiple features simultaneously, often using techniques like
CART (Classification and Regression Trees).
 Advantages: Can capture more complex relationships between features compared to univariate trees.

Summary

 Clustering techniques like k-means, EM, and hierarchical clustering help identify patterns in data
without prior labels, useful for exploratory analysis and preprocessing for supervised learning.
 Decision Trees are powerful tools for both classification and regression, employing simple, interpretable
structures to model complex relationships in the data. Techniques such as pruning and rule extraction
enhance their usability.