Web Mining & Text Mining

Web Mining & Text Mining

• 1.
  Web Mining &Text Mining Prepared by : Sharma Hemant hemantbeast@gmail.com
• 2.
  Web Mining
• 3.
  Web Mining  WebMining is the application of data mining techniques to extract knowledge from web data such as Web content, Web structure and Web usage data.  It is the process of discovering the useful and previously unknown information from the web data.  Web data is :- • Web content :- text, images, records, etc. • Web structure :- hyperlinks, tags, etc. • Web usage :- http logs, app server logs, etc.
• 4.
  Web Mining
• 5.
  Web Content Mining Web content mining performed by extracting useful information from the content of a web page/site.  It includes extraction of structured data/information from web pages, identification, match, and integration of semantically similar data.  The type of web content may consist of text, image, audio, video, etc. It is also know as text mining.  It uses the Natural Language Processing and Information Retrieval techniques for mining the data.
• 6.
  Web Structure Mining The structure of a typical Web graph consists of Web pages as nodes, and hyperlinks as edges connecting between two related pages.  Web structure mining is the process of discovering structure information from the web. • This type of mining can be performed either at the (intra-page) document level or the (inter-page) hyperlink level. • The research at the hyperlink level is also called Hyperlink Analysis.
• 7.
  Web Structure Terminology Web-graph : A directed graph that represents the Web.  Node : Each Web page represents a node of the Web-graph.  Link : Each hyperlink on the Web is a directed edge of the Web-graph.  In-degree : The number of distinct links that point to a node.  Out-degree : The number of distinct links originating at a node that point to other nodes.
• 8.
   Directed Path: It is a sequence of links, starting from a node say r that can be followed to reach another node say t.  Shortest Path : The path with the shortest length out of all the paths between nodes p and q.  Diameter : It is the maximum of all the shortest paths between a pair of nodes p and q, for all pairs of nodes p and q in the Web-graph. Web Structure Terminology
• 9.
  Web Usage Mining A Web is a collection of inter-related files on one or more Web Servers.  Discovery of meaningful patterns from data generated by client-server transaction on one or more Web localities.  Typical Sources of Data : • Automatically generated data stored in server access logs, referrer logs, agent logs, and client-side cookies. • User profiles. • Metadata : page attributes, content attributes, usage data.
• 10.
   Web servers,Web proxies, and client application can quite easily capture Web Usage data.  Web Server Log : It is a file that is created by the server to record all the activities it performs.  For ex: When a user enters URL into the browsers address bar or requests by clicking on a link.  The page request sent to web server maintains the following info. in its log like Information about URL, Whether the request was successful, Users IP address, time and date, etc. Web Usage Mining
• 11.
  Text Mining
• 12.
  Text Mining  Theobjective of Text Mining is to exploit information contained in textual documents in various ways, including discovery of patterns and trends in data, associations among entities, predictive rules, etc.  The results can be important both for : • The analysis of the collection, and • Providing intelligent navigation and browsing methods.
• 13.
  Text Mining Workflow
• 14.
  Data Mining vsText Mining  Both seek novel and useful pattern.  Both are semi-automated process.  Difference is the nature of the data: • Structured versus Unstructured data • Structured data: databases • Unstructured data: word docs, pdf files, xml files, and so on  Text mining – first, impose structure to the data, then mine the structured data.
• 15.
  Technology premise ofText Mining  Summarization : It is a process of making summary of any document containing large amount of information while theme or main idea of document is maintained.  Information Extraction : It utilizes relations within the text. It uses pattern matching for it.  Categorization : It is a supervised learning technique which places the document according to content. Document categorization is largely used in libraries.  Visualization : It is computer graphic effect to represent information and revealing relationships.
• 16.
   Clustering :It is a document’s textual similarity based unsupervised technique which is used by data analysis to divide the text into mutually exclusive groups.  Question Answering : Natural language queries or questions answering is responsible to decide a way find a more suitable answer for particular question.  Sentiment Analysis : It is also known as opinion mining is configured of user’s emotion, mostly into several classes which are positive, negative, neutral and mixed. It is mainly used to get people’s view or attitude towards anything which includes services and products. Technology premise of Text Mining