Downloaded 213 times
Uploaded byFrank Kelly
Hierarchical clustering in Python and beyond
The document discusses hierarchical clustering techniques in Python, covering theory, practical applications, and relevant methodologies for data analysis. It highlights the importance of various clustering methods, metrics, and considerations for achieving efficient results, particularly in relation to search keyword analysis. Additionally, it emphasizes the role of visualization tools and the necessity of preprocessing data for effective clustering outcomes.
In this document
Powered by AI
Overview of hierarchical clustering methods, theories, visualizations, and use cases.
Clustering as an unsupervised learning tool to group similar entities based on data similarity.
Origins of clustering in anthropology and psychology with diverse applications.
Clustering can be used for exploratory analysis and as part of supervised learning pipelines.
Important factors in clustering including criteria, space, separation types, and similarity measures.
Exploration of clustering search keywords for better website alignment and opportunity identification.
Need for visual insights in clustering high-dimensional data and recommended methods for text clustering.
Introduction to agglomerative and divisive hierarchical clustering techniques.
Various linkage types in agglomerative clustering including single, complete, and Ward’s criterion.
Different measures of similarity used in clustering such as Euclidean and Cosine similarity.
Preparation of text clustering including keyword aggregation and tokenization.
Challenges and techniques for reducing dimensionality in high-dimensional clustering datasets.
Methods for assessing cluster quality through metrics like purity and adjusted Rand index.
Integration of clustering within Elasticsearch and introduction to Lingo 3G algorithm for clustering.
Drawbacks of hierarchical clustering and challenges including scalability and irreversibility of merges.
Resources for learning more about clustering techniques and why working within databases is beneficial.
More Related Content
![[Karger+ NIPS11] Iterative Learning for Reliable Crowdsourcing Systems](https://cdn.slidesharecdn.com/ss_thumbnails/karger-croudsourcing-nips11-120408005300-phpapp01-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Similar to Hierarchical clustering in Python and beyond
Hierarchical clustering in Python and beyond
- 1. Hierarchical clustering in Python& elsewhere For @PyDataConf London, June 2015, by Frank Kelly Data Scientist, Engineer @analyticsseo @norhustla
- 2. Hierarchical Clustering Theory Practice Visualisation Origins& definitions Methods & considerations Hierachical theory Metrics & performance My use case Python libraries Example Static Interactive Further ideas All opinions expressed are my own
- 3. Who am I? Allopinions expressed are my own
- 4.
- 5. Clustering is anunsupervised learning problem "SLINK-Gaussian-data" by Chire - Own work. Licensed under CC BY-SA 3.0 via Wikimedia Commons - https://commons.wikimedia.org/wiki/File:SLINK-Gaussian-data.svg#/media/File:SLINK-Gaussian-data.svg based on some notion of similarity. whereby we aim to group subsets of entities with one another
- 6.
- 7. Diverse applications Attribution: stackoverflow, wikipedia, scikit-learn.org, http://www.poolparty.biz/
- 8. Two main purposes Exploratory analysis –standalone tool (Data mining) As a component of a supervised learning pipeline (in which distinct classifiers or regression models are trained for each cluster). (Machine Learning)
- 9. Clustering considerations Partitioning criteria (single / multilevel) Separation Exclusive / non- exclusive Clustering space (Full-space / sub-space) Similarity measure (distance / connectivity)
- 10. Use case: searchkeywords RD P P P KW KW KW KW KW CP CP KW KW KW The competition! KW KW CP CD You Opportunity! CD = Competing domains CP = Competitor’s pages RD = Ranking domain P = Your page KW = Keyword
- 11.
- 12. Use case: searchkeywords KW…so we have found 100,000 new ‘s – now what? How do we summarise and present these to a client?
- 13. Clients’ questions… • Dosearch categories in general align with my website structure? • Which categories of opportunity keywords have the highest search volume, bring the most visitors, revenue etc.? • Which keywords are not relevant?
- 14.
- 15. Requirements • Need: visualinsights; structure • Allow targeting of problem in hand • May develop into a semi- supervised solution
- 16. • High-dimensional andsparse data set • Values correspond to word frequencies • Recommended methods include: hierarchical clustering, Kmeans with an appropriate distance measure, topic modelling (LDA, LSI), co-clustering Options for text clustering?
- 17.
- 18. 2 types Agglomerative Divisive Deterministicalgorithms! Attribution: Wikipedia
- 19. Agglomerative Start with many “singleton”clusters … Merge 2 at a time continuously … Build a hierarchy Divisive Start with a huge “macro” cluster … Iteratively split into 2 groups … Build a hierarchy
- 20. Agglomerative method: Linkage types •Single (similarity between most similar – based on nearest neighbour - two elements) • Complete (similarity between most dissimilar two elements) Attribution: https://www.coursera.org/course/clusteranalysis
- 23. Agglomerative method: Linkage types Averagelink ( avg. of similarity between all inter-cluster pairs ) Computationally expensive (Na*Nb) Trick: Centroid link (similarity between centroid of two clusters) Attribution: https://www.coursera.org/course/clusteranalysis
- 24. Ward’s criterion • Minimisea function: total in-cluster variance • As defined by, e.g.: • Once merged, then the SSE will increase (cluster becomes bigger) by: https://en.wikipedia.org/wiki/Ward's_method
- 25. Divisive clustering • Top-downapproach • Criterion to split: Ward’s criterion • Handling noise: Use a threshold to determine the termination criteria Attribution: https://www.coursera.org/course/clusteranalysis
- 26. Similarity measures This willcertainly influence the shape of the clusters! • Numerical: Use a variation of the Manhattan distance (e.g. City block, Euclidean) • Binary: Manhattan, Jaccard co-efficient, Hamming • Text: Cosine similarity.
- 27. Cosine similarity Represent adocument by a bag of terms Record the frequency of a particular term (word/ topic/ phrase) If d1 and d2 are two term vectors, …can thus calculate the similarity between them Attribution: https://www.coursera.org/course/clusteranalysis
- 29. Gather word documents= keyword phrases
- 30. Aggregate search wordswith URL “words”
- 31. Text clustering: preparations • Addfeatures where possible o I added URL words to my word set • Stem words o Choose the right stemmer – too severe can be bad • Stop words o NLTK tokeniser o Scikit learn TF-IDF tokeniser • Low frequency cut-off o 2 => words appearing less than twice in whole corpus • High frequency cut-off o 0.5 => words that appear in more than 50% of documents • N-grams o Single words, bi-grams, tri-grams • Beware of foreign languages o Separate datasets if possible
- 32.
- 33. Dimensionality • Get asparse matrix o Mostly zeros • Reduce the number of dimensions o PCA o Spectral clustering • The “curse” of dimensionality
- 35.
- 36.
- 37.
- 38. Assess the qualityof your clusters • Internal: Purity, completeness & homogeneity • External: Adjusted Rand index, Normalised Information index
- 39.
- 40.
- 41. Life on theinside: Elasticsearch • Why not perform pre-processing and clustering inside elasticsearch? • Document store • TF-IDF and other • Stop words • Language specific analysers
- 42. Elasticsearch - try it! - • https://www.elastic.co/ • NoSQL document store • Aggregations and stats • Fast, distributed • Quick to set up
- 43.
- 44. Lingo 3G algorithm •Lingo 3G: Hierarchical clustering off-the-shelf • Built-in part of speech (POS) • User-defined word/synonym/label dictionaries • Built-in stemmer / word inflection database • Multi-lingual support, advanced tuning • Commercial: costs attached http://download.carrotsearch.com/lingo3g/manual/#section.es http://project.carrot2.org/algorithms.html
- 45. Elasticsearch with clustering –Utopia? Carrot2’s Lingo3G in action : http://search.carrot2.org/stable/search Foamtree visualisation example Visualisation of hierarchical structure possible for large datasets via “lazy loading” http://get.carrotsearch.com/foamtree/demo/demos/large.html
- 46. Limitations of hierarchical clustering •Can’t undo what’s done (divisive method, work on sub clusters, cannot re-merge). Even true for agglomerative (once merged will never split it again) • Every split or merge must be refined • Methods may not scale well, checking all possible pairs, complexity goes high There are extensions: BIRCH, CURE and CHAMELEON
- 47. Thank you! A decentintroductory course to clustering; https://www.coursera.org/course/clusteranalysis Hierarchical (agglomerative) clustering in Python: http://scikit- learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html Recent (ish) relevant Kaggle challenge: https://www.kaggle.com/c/lshtc Visualisation: http://carrotsearch.com/foamtree-overview Clustering elsewhere (Lingo, Lingo3G) with Carrot2:http://download.carrotsearch.com/ Elasticsearch: https://www.elastic.co/ Analytics SEO: http://www.analyticsseo.com/ Me: @norhustla / frank.kelly@cantab.net Attribution: http://wynway.com/
- 48. Extra slide: Whywork inside the database? 1. Sharing data (management of) Support concurrent access by multiple readers and writers 2. Data Model Enforcement Make sure all applications see clean, organised data 3. Scale Work with datasets too large to fit in memory (over a certain size, need specialised algorithms to deal with the data -> bottleneck) The database organises and exposes algorithms for you conveniently 4. Flexibility Use the data in new, unanticipated ways -> anticipate a broad set of ways of accessing the data