Downloaded 181 times
![Ugly Demo IV: Frequent Pattern MiningData: http://fimi.cs.helsinki.fi/data/./mahout fpg -i <PATH>/content/freqitemset/accidents.dat -o patterns -k 50 -method mapreduce -g 10 -regex [\ ] ./mahout seqdump --seqFile patterns/fpgrowth/part-r-00000](https://image.slidesharecdn.com/intro-mahout-110529200909-phpapp01/85/mahout-introduction-24-320.jpg)
![Ugly Demo IV: Frequent Pattern MiningData: http://fimi.cs.helsinki.fi/data/./mahout fpg -i <PATH>/content/freqitemset/accidents.dat -o patterns -k 50 -method mapreduce -g 10 -regex [\ ] ./mahout seqdump --seqFile patterns/fpgrowth/part-r-00000](https://image.slidesharecdn.com/intro-mahout-110529200909-phpapp01/75/mahout-introduction-24-2048.jpg)
Uploaded bychanggeng Zhang
mahout introduction
This document introduces Apache Mahout, an open source machine learning library. It discusses common machine learning use cases like recommendations, classification, and clustering. It explains how Mahout implements scalable machine learning algorithms using Apache Hadoop. Finally, it provides examples of using Mahout's recommender systems, topic modeling, clustering and frequent pattern mining capabilities.
mahout introduction
- 1. Introduction to ScalableMachine Learning with Apache Mahout Grant IngersollFebruary 15, 2010
- 2. IntroductionYouMachine learning experience?BusinessIntelligence?Natural Lang. Processing?ApacheHadoop?MeCo-founder Apache MahoutApache Lucene/Solr committerCo-founder Lucid Imagination
- 3. TopicsWhat is MachineLearning?ML Use CasesWhat is Mahout?What can I do with it right now?Where’s Mahout headed?
- 4. Amazon.comWhat is MachineLearning?Google News
- 5. Really it’s…“Machine Learningis programming computers to optimize a performance criterion using example data or past experience”Intro. To Machine Learning by E. AlpaydinSubset of Artificial IntelligenceLots of related fields:Information RetrievalStatsBiologyLinear algebraMany more
- 6. Common Use CasesRecommendfriends/dates/productsClassify content into predefined groupsFind similar content based on object propertiesFind associations/patterns in actions/behaviorsIdentify key topics in large collections of textDetect anomalies in machine outputRanking search resultsOthers?
- 7.
- 8. Getting Started withMLGet your dataDecide on your features per your algorithmPrep the dataDifferent approaches for different algorithmsRun your algorithm(s)Lather, rinse, repeatValidate your resultsSmell test, A/B testing, more formal methods
- 9. Apache Mahouthttp://dictionary.reference.com/browse/mahoutAn ApacheSoftware Foundation project to create scalable machine learning libraries under the Apache Software LicenseWhy Mahout?Many Open Source ML libraries either:Lack CommunityLack Documentation and ExamplesLack ScalabilityLack the Apache License ;-)Or are research-oriented
- 10. Focus: Machine LearningApplicationsExamplesRecommendersClusteringClassificationFreq.PatternMiningGeneticMathVectors/Matrices/SVDUtilitiesLucene/VectorizerCollections (primitives)Apache HadoopSee http://cwiki.apache.org/confluence/display/MAHOUT/Algorithms
- 11. Focus: ScalableGoal: Beas fast and efficient as the possible given the intrinsic design of the algorithmSome algorithms won’t scale to massive machine clustersOthers fit logically on a Map Reduce framework like Apache HadoopStill others will need other distributed programming modelsBe pragmaticMost Mahout implementations are Map Reduce enabledWork in Progress
- 12. Prepare Data fromRaw contentData Sources:Lucene integrationbin/mahout lucenevector …Document Vectorizerbin/mahout seqdirectory …bin/mahout seq2sparse …ProgrammaticallySee the Utils module in MahoutDatabaseFile system
- 13. RecommendationsExtensive framework forcollaborative filteringRecommendersUser basedItem basedOnline and Offline supportOffline can utilize HadoopMany different Similarity measuresCosine, LLR, Tanimoto, Pearson, others
- 14. ClusteringDocument levelGroup documentsbased on a notion of similarityK-Means, Fuzzy K-Means, Dirichlet, Canopy, Mean-ShiftDistance MeasuresManhattan, Euclidean, otherTopic Modeling Cluster words across documents to identify topicsLatent Dirichlet Allocation
- 15. CategorizationPlace new itemsinto predefined categories:Sports, politics, entertainmentMahout has several implementationsNaïve BayesComplementary Naïve BayesDecision Forests
- 16. Freq. Pattern MiningIdentifyfrequently co-occurrent itemsUseful for:Query RecommendationsApple -> iPhone, orange, OS XRelated product placement“Beer and Diapers”http://www.amazon.com
- 17. EvolutionaryMap-Reduce ready fitnessfunctions for genetic programmingIntegration with Watchmakerhttp://watchmaker.uncommons.org/index.phpProblems solved:Traveling salesmanClass discoveryMany others
- 18. How To: RecommendersData:Users (abstract)Items (abstract)Ratings (optional)Load the data modelAsk for Recommendations:User-UserItem-Item
- 19. Ugly Demo IGroupLens Data: http://www.grouplens.orghttp://lucene.apache.org/mahout/taste.html#demohttp://localhost:8080/RecommenderServlet?userID=1&debug=trueIn other words: the reason why I work on servers, not UIs!
- 20. How to: CommandLineMost algorithms have a Driver programShell script in $MAHOUT_HOME/bin helps with most tasksPrepare the DataDifferent algorithms require different setupRun the algorithmSingle NodeHadoopPrint out the resultsSeveral helper classes: LDAPrintTopics, ClusterDumper, etc.
- 21. Ugly Demo II- PrepData Set: Reutershttp://www.daviddlewis.com/resources/testcollections/reuters21578/Convert to Text via http://www.lucenebootcamp.com/lucene-boot-camp-preclass-training/Convert to Sequence File:bin/mahout seqdirectory –input <PATH> --output <PATH> --charset UTF-8Convert to Sparse Vector:bin/mahout seq2sparse --input <PATH>/content/reuters/seqfiles/ --norm 2 --weight TF --output <PATH>/content/reuters/seqfiles-TF/ --minDF 5 --maxDFPercent 90
- 22. Ugly Demo II:Topic ModelingLatent Dirichlet Allocation./mahout lda --input <PATH>/content/reuters/seqfiles-TF/vectors/ --output <PATH>/content/reuters/seqfiles-TF/lda-output --numWords 34000 –numTopics 10./mahout org.apache.mahout.clustering.lda.LDAPrintTopics --input <PATH>/content/reuters/seqfiles-TF/lda-output/state-19 --dict <PATH>/content/reuters/seqfiles-TF/dictionary.file-0 --words 10 --output <PATH>/content/reuters/seqfiles-TF/lda-output/topics --dictionaryTypesequencefileGood feature reduction (stopword removal) required
- 23. Ugly Demo III:ClusteringK-MeansSame Prep as UD II, except use TFIDF weight./mahout kmeans --input <PATH>/content/reuters/seqfiles-TFIDF/vectors/part-00000 --k 15 --output <PATH>/content/reuters/seqfiles-TFIDF/output-kmeans --clusters <PATH>/content/reuters/seqfiles-TFIDF/output-kmeans/clustersPrint out the clusters: ./mahout clusterdump --seqFileDir <PATH>/content/reuters/seqfiles-TFIDF/output-kmeans/clusters-15/ --pointsDir <PATH>/content/reuters/seqfiles-TFIDF/output-kmeans/points/ --dictionary <PATH>/content/reuters/seqfiles-TFIDF/dictionary.file-0 --dictionaryTypesequencefile --substring 20
- 24. Ugly Demo IV:Frequent Pattern MiningData: http://fimi.cs.helsinki.fi/data/./mahout fpg -i <PATH>/content/freqitemset/accidents.dat -o patterns -k 50 -method mapreduce -g 10 -regex [\ ] ./mahout seqdump --seqFile patterns/fpgrowth/part-r-00000
- 25. What’s Next?0.3 releasevery soonParallel Singular Value Decomposition (Lanczos)Stabilize API’s for 1.0 releaseBenchmarkingGoogle Summer of Code?More Algorithmshttp://cwiki.apache.org/MAHOUT/howtocontribute.html
- 26. ResourcesSlides and FullDetails of Demos at:http://lucene.grantingersoll.com/2010/02/13/intro-to-mahout-slides-and-demo-examples/More Examples in Mahout SVN in the examples directory
- 27.
- 28. Resources“Mahout in Action”by Owen and Anil“Introducing Apache Mahout”http://www.ibm.com/developerworks/java/library/j-mahout/“Programming Collective Intelligence” by Toby Segaran“Data Mining - Practical Machine Learning Tools and Techniques” by Ian H. Witten and Eibe Frank
- 29. ReferencesHAL: http://en.wikipedia.org/wiki/File:Hal-9000.jpgTerminator: http://en.wikipedia.org/wiki/File:Terminator1984movieposter.jpgMatrix:http://en.wikipedia.org/wiki/File:The_Matrix_Poster.jpgGoogle News: http://news.google.comAmazon.com: http://www.amazon.comFacebook: http://www.facebook.comMahout: http://lucene.apache.org/mahoutBeer and Diapers: http://www.flickr.com/photos/baubcat/2484459070/http://www.theregister.co.uk/2006/08/15/beer_diapers/DMOZ: http://www.dmoz.org