Downloaded 36 times
![TWEET ANALYSIS 3
We want to break up the tweet into the following
parts:
{
'text': ['PBS addresses mental health needs in the aftermath of the Haiti
earthquake'],
'hashtags': ['#Haiti', '#health', '#earthquake'],
'names': ['@directrelief', '@PIH', '@NewsHour'],
'urls': ['http://bit.ly/bNhyK6'],
}](https://image.slidesharecdn.com/swiftoverview2011-110318014336-phpapp01/85/SwiftRiver-2011-Overview-30-320.jpg)
![TWEET ANALYSIS 3
We want to break up the tweet into the following
parts:
{
'text': ['PBS addresses mental health needs in the aftermath of the Haiti
earthquake'],
'hashtags': ['#Haiti', '#health', '#earthquake'],
'names': ['@directrelief', '@PIH', '@NewsHour'],
'urls': ['http://bit.ly/bNhyK6'],
}](https://image.slidesharecdn.com/swiftoverview2011-110318014336-phpapp01/75/SwiftRiver-2011-Overview-30-2048.jpg)
More Related Content
Similar to SwiftRiver 2011 Overview
SwiftRiver 2011 Overview
- 1. SWIFT RIVER 2011 Jon Gosier, Director of Product http://swiftly.org @swiftriver @jongos An Ushahidi Initiative
- 2. Initial development beganduring the Haiti earthquakes, one of Ushahidi’s largest deployments to date. Objective became to offer smart tools for curating real- time data of all types (Email, Twitter, SMS, Web feeds).
- 3. PLATFORM GOALS ‣ Democratizeaccess to intelligence tools ‣ Structure unstructured data feeds ‣ Data-mine overwhelming realtime datasets ‣ Surface signal & suppress noise ‣ Identify and rate authoritative users & sources ‣ Easy to use tools & applications for curating data on the user’s terms
- 4. “It’s not informationoverload. It’s filter failure.” - Clay Shirky
- 5. Sweeper - UserInterface
- 6. The Brain asan API ‣ Breaks content (data) into pieces ‣ Analyzes pieces separately ‣ Conditionally prioritizes ‣ Learns from experience ‣ Processing is distributed ‣ Recombination of pieces
- 7. APIs Applications ‣ Tagging API - parses text and adds taxonomy ‣ Location API - detects origin location of content ‣ Sweeper - sweep, structure and sort ‣ Influence API - measures influence of content online realtime data-streams ‣ Reputation API - stores information about user ‣ SwiftMeme - meme/keyword tracker, behavior content discovery ‣ Duplication Filter API - receives feeds and filters out ‣ SwiftMail - sort email by relevance duplicate content to cut down on retweets
- 10.
- 11. Queensland - ABCAustralia Deployment
- 12. RIVER ID Global Trustand Reputation Server Web Services
- 13. WHAT IS RIVERID? • Opt-in product for Ushahidi deployments • Collect information on all contributors • Use contributions to build trust profile • Use trust profile to help validate information in the future • Global trust bank built on OAUTH standards
- 15. REVERBERATIONS Measuring Influence Web Services
- 16. INFORMATION REVERBERATES • Good information and Bad information spread the same • Reverberations tracks influences • Breadcrumb trails for information and content
- 20. AUTO-TAGGING SILCC: SwiftRiver LanguageComputation Core Web Services
- 21. WHAT IS SILCC? •SwiftLanguage Computation Component •One of the SwiftRiver Web Services •Open Web API •Semantic Tagging of Short Text •Natural Language Processing •Multilingual •Multiple sources (twitter, email, SMS, blogs etc) •Active Learning capability
- 22. Swiftriver SiLCC Dataflow SiSLS Content Items coming from the SiSLS have where Swiftriver Source SiSLS integrations is enabled global trust values Library Service added to the object model. SiLCC Swiftriver Language An API key is sent along with the text to ensure that the SiLCC is not open to any malicious usage. Computational Core The text of the content is sent to the SiLCC. There is still a bit of ambiguity around what the NLP should extract from the text but at its most simple, Using NLP, the SiLCC all the nouns would be a good start. extracts Nouns and other keywords from the text. The SiLCC send back The lists of tags sent back from the SiLCC can be a list of tags that are added to the Content Item along with any that were added to the extracted from the source data by the parser. Content Item SLISa Although the NLP tags have now been applied, the SLISa is now responsible for applying instance Swiftriver Language specific tagging corrections. Improvement Service
- 23. OUR GOALS •Simple Taggingof short snippets of text •Rapid tagging for high volume environments •Simple API, easy to use •Learns from user feedback •Routing of messages to upstream services •Semantic Classification •Sorts rapid streams into buckets •Clusters like messages •Visual effects •Cross-referencing
- 24. WHAT IT’S NOT •Doesnot do deep analysis of text •Only identifies words within original text
- 25. HOW DOES ITWORK? •Step 1: Lexical Analysis •Step 2: Parsing into constituent parts •Step 3: Part of Speech tagging •Step 4: Feature extraction •Step 5: Compute using feature weights •Lets examine each one in turn...
- 26. STEP 1: LEXICALANALYSIS •For news headlines, email subjects this is trivial, just split on spaces. •For Twitter this is more complex...
- 27. TWEET ANALYSIS •Tweets aresurprisingly complex •Only 140 characters but many features •Emergent features from community (e.g. hashtags) •Lets take a look at a typical tweet...
- 28. TWEET ANALYSIS Thetypical Tweet: “RT @directrelief: RT @PIH: PBS @NewsHour addresses mental health needs in the aftermath of the #Haiti earthquake #health #earthquake... http://bit.ly/bNhyK6” •RT indicates a “re-tweet” •@name indicates who the original tweeter was •Multiple embedded retweets •Hashtags (e.g. #Haiti) can play two roles, as a tag and as part of the sentence
- 29. TWEET ANALYSIS 2 •Twoor more hashtags within a tweet (e.g. #health and #earthquake) •Continuation dots “...” indicates that there was more text that didn’t fit into the 140 limit somewhere in it’s history •Urls many tweets contain one or more urls As we can see this simple tweet contains no less than 7 different features and that’s not all!
- 30. TWEET ANALYSIS 3 Wewant to break up the tweet into the following parts: { 'text': ['PBS addresses mental health needs in the aftermath of the Haiti earthquake'], 'hashtags': ['#Haiti', '#health', '#earthquake'], 'names': ['@directrelief', '@PIH', '@NewsHour'], 'urls': ['http://bit.ly/bNhyK6'], }
- 31. TWEET ANALYSIS 4 Why do we want to break up the tweet into parts (parsing)? •Because we want to further process the grammatically correct english text •Part of speech tagging would otherwise be corrupted by words it cannot recognize (e.g. urls, hashtags, @names etc.) •We want to save the hashtags for later use •Many of the features are irrelevant to the task of identifying tags (e.g. dots, punctuation, @name, RT)
- 32. TWEET ANALYSIS 5 •Wenow take the “text” portion of the tweet and perform part of speech tagging on it •After part of speech tagging, we perform feature extraction •Features are now passed through the keyword classifier which returns a list of keywords / tags •Finally we combine these tags with the hashtags we saved earlier to give the complete tag set
- 33. HEADLINE AND EMAIL SUBJECT ANALYSIS •This is much simpler to do •Its a subset of the steps in Tweet Analysis •There is no parsing since there are no hashtags, @names etc.
- 34. FEATURE EXTRACTION • Forthe active learning algorithm we need to extract features to use in classification • These features should be subject/domain independent • We therefore never use the actual words as features • This would for example give artificially high weights to words such as “earthquake” • We don't want these artificial weights as we can’t foresee future disasters and we want to be as generic with classification as possible • The use of training sets does allow for domain customization if where necessary
- 35. FEATURE EXTRACTION • Capitalizationof individual words: Either first caps, or all caps, this is an important indicator of proper nouns or other important words that make good tag candidates • Position in text: Tags seem to have a greater preponderance near the beginning of text • Part of Speech: Nouns and proper nouns are particularly important but so are some adjectives and adverbs • Capitalization of entire text: sometimes the whole text is capitalized and this should reduce overall weighting of other features • Length of the text: In shorter texts the words are more likely to be tags • The parts of speech of previous and next words (effectively this means we are using trigrams; or a window of 3)
- 36. TRAINING • Requires userreviewed examples • Lexical analysis, parsing and feature extraction on the examples • Multinomial naïve Bayes algorithm • NB: The granularity we are classifying is at the word level • For each word in the text, we classify it as either a keyword or not • This has pleasant side effect of providing several training examples from each user reviewed text • Even with less than 50 reviewed texts the results are comparable to the simple approach of using nouns only
- 37. ACTIVE LEARNING •The APIalso provides a method for users to send back corrected text •The corrected text is saved and then used in the next iteration of training •User may optionally specify a corpus for the example to go into •Training can be performed using any combination of corpora
- 38. DEVELOPER FRIENDLY •Two levelsof API, the web API and the internal Python API •Either one may be used but most users will use the web API •Design is highly modular and maintainable •For very rapid backend processing the native Python API can be used
- 39. PYTHON CLASSES Most ofthe classes that make up the library are divided into three types: 1) Tokenizers 2) Parsers 3) Taggers All three types have consistent API's and are interchangeable.
- 40. PYTHON API •A taggercalls a parser •A parser calls a tokenizer •Output of the tokenizer goes into the parser •Output of the parser goes into the tagger •Output of the tagger goes into the user!
- 41. CLASSES • BasicTokenizer –This is used for splitting basic (non-tweet) text into individual words • TweetTokenizer – This is used to tokenize a tweet, it may also be used to tokenize plain text since plain text is a subset of tweets • TweetParser – Calls the TweetTokenizer and the parses the output (see previous example) • TweetTagger – Calls the TweetTokenizer and then tags the output of the text part and adds the hashtags • BasicTagger – Calls the BasicTokenizer and then tags the text, should only be used for non-tweet text, uses simple Part of Speech to identify tags • BayesTagger – Same as BasicTagger but uses weights from the naïve Bayes training algorithm
- 42. DEPENDANCIES •Part of speechtagging is currently performed by the Python NLTK •The Web API uses the Pylons web framework
- 43. CURRENT STATUS •Tag methodof API is ready for use, individual deployments can choose between using the BasicTagger or the BayesTagger •Tell method (for user feedback) will be ready by the time you read this! •Training is possible on corpora of tagged data in .csv format (see examples in distribution)
- 44. CURRENT LIMITATIONS •Only Englishtext is supported at the moment •Tags are always one of the words in the supplied text ie they can never be a word not in the supplied text •Very few training examples exist at the moment
- 45. GEO DICT LOCATION DISAMBIGUATION Web Services
- 46. WHAT IS GEODICT? • For auto-mapping data • Reverse lookup lat/lon from place names • Works with data from Twitter, Email, RSS, SMS • SVM or Naive Bayes/Fisher Classification • Database of global place-names corresponding lat/lon
- 47. ITEM CLASSIFICATION • Feature Extraction: Bag of Words, String Kernels • Higher Level Features: Topic Modeling • Linguistic pre-processing: lemmatization, stemming • Natural Language Processing • Named Entity Recognition • Multi-class classification: One-vs.-One, One-vs.-All
- 48. SWIFT RIVER Jon Gosier,Director of Product http://swiftly.org @swiftriver @jongos An Ushahidi Initiative