Downloaded 183 times
![Association analysis(example)
34
Item frequently purchased together
buys(X, ”computer”) =>buys(X, ”software”)
[support=1%, confidence=50%]
X - a variable representing a customer
A confidence or certainty – 50%(chance)
1%(under analysis)
Association rule- with single-dimension association rules
“computer => software[1%,50%]”.
Age(X,”20..29”) ^ income(X,”40K..49K”)=>buys(X ,”laptop”)
[support=2%, confidence=60%] (Multidimensional association rule)](https://image.slidesharecdn.com/introduction-140222064140-phpapp01/85/Introduction-to-DataMining-34-320.jpg)
![Association analysis(example)
34
Item frequently purchased together
buys(X, ”computer”) =>buys(X, ”software”)
[support=1%, confidence=50%]
X - a variable representing a customer
A confidence or certainty – 50%(chance)
1%(under analysis)
Association rule- with single-dimension association rules
“computer => software[1%,50%]”.
Age(X,”20..29”) ^ income(X,”40K..49K”)=>buys(X ,”laptop”)
[support=2%, confidence=60%] (Multidimensional association rule)](https://image.slidesharecdn.com/introduction-140222064140-phpapp01/75/Introduction-to-DataMining-34-2048.jpg)
Uploaded bySAS SNDP YOGAM COLLEGE,KONNI
Introduction to DataMining
1) Data mining involves extracting hidden patterns from large datasets to discover useful information. It is an interdisciplinary field drawing from statistics, machine learning, database technology and more. 2) The overall goal is to extract information and transform it into an understandable structure. This includes data cleaning, integration, selection, transformation, mining patterns, and evaluating/presenting the results. 3) Data mining is used for applications like market analysis, risk analysis, fraud detection and more, across domains like business, science, health, and society. It has the potential to provide insights from vast amounts of accumulated data.
More Related Content
Similar to Introduction to DataMining
Introduction to DataMining
- 1.
- 2. 2 A young, fastgrowing and promising field
- 3. INTRODUCTION 3 Data mining (theanalysis step of the "Knowledge Discovery and Data Mining" process, or KDD) Extracting hidden information An interdisciplinary subfield of computer science The computational process of discovering patterns in large data sets Involving methods at the intersection of Artificial intelligence, Machine learning, Statistics, and Database systems.
- 4. INTORODUCTION(CONTD..) 4 The overall goalof the data mining process is to extract information from a data set and transform it into an understandable structure for further use. Aside from the raw analysis step, it involves • database and data management aspects • • data pre-processing model inference considerations complexity considerations, post-processing of discovered structures, visualization, and online updating.
- 5. Why Data Mining? 5 TheExplosive Growth of Data: from terabytes to petabytes Eg: Global backbone telecommunication network carry tens of petabytes everyday (1024 Gigabytes = 1 Terabyte)( 1024 Terabytes = 1 Petabyte) Data collection and data availability Automated data collection tools, database systems, Web, computerized society Major sources of abundant data Business: Web, e-commerce, transactions, stocks, … Science: Remote sensing, bioinformatics, scientific simulation, … Society and everyone: news, digital cameras,…
- 6. Why Data Mining? 6 “Necessityis the mother of invention” - Data mining—Automated analysis of massive data sets
- 7. What Motivated DataMining? 7 We are drowning in data, but starving for knowledge!
- 8. Evolution of Database Technology 8 Datamining can be viewed as a result of natural evolution of IT 1960s: 1970s: Data collection, database creation and network DBMS Relational data model, relational DBMS implementation 1980s: RDBMS, advanced data models (extended-relational, OO, deductive, etc.) Application-oriented DBMS (spatial, scientific, engineering, etc.)
- 9. Evolution of DatabaseTechnology 9 1990s: Data mining, data warehousing, multimedia databases, and Web databases 2000s Stream data management and mining Data mining and its applications Web technology (XML, data integration) and global information systems
- 10.
- 11. What Is DataMining? 11 Data mining (knowledge discovery from data) Extraction of interesting (non-trivial, implicit, previously unknown and potentially useful) patterns or knowledge from huge amount of data Alternative names Knowledge discovery (mining) in databases (KDD), knowledge extraction, data/pattern analysis, data archeology, data dredging, information harvesting, business intelligence, etc. Watch out: Is everything “data mining”? Simple search and query processing (Deductive) expert systems
- 12. Data Mining: Confluenceof Multiple Disciplines 12 Database Technology Machine Learning Pattern Recognition Statistics Data Mining Algorithm Visualization Other Disciplines
- 13. Knowledge Discovery (KDD)Process 13 Data mining—core of knowledge discovery process Pattern Evaluation Data Mining Task-relevant Data Data Warehouse Data Cleaning Data Integration Databases Selection
- 14. Knowledge Process 14 1. 2. 3. 4. 5. 6. 7. Data cleaning– to remove noise and inconsistent data Data integration – to combine multiple source Data selection – to retrieve relevant data for analysis Data transformation – to transform data into appropriate form for data mining Data mining- An essential process where intelligent methods are applied to extract data patterns Pattern Evaluation-Identify truly interesting patterns representing knowledge based on interestingness measure Knowledge presentation-visualization and representation techniques
- 15. Example: A WebMining Framework 15 Web mining usually involves Data cleaning Data integration from multiple sources Warehousing the data Data cube construction Data selection for data mining Data mining Presentation of the mining results Patterns and knowledge to be used or stored into knowledge-base
- 16. Data Mining inBusiness Intelligence Increasing potential to support business decisions End User Decision Making Business Analyst Data Presentation Visualization Techniques Data Mining Information Discovery Data Analyst Data Exploration Statistical Summary, Querying, and Reporting Data Preprocessing/Integration, Data Warehouses Data Sources Paper, Files, Web documents, Scientific experiments, Database Systems 16 DBA
- 17. KDD Process: ATypical View from ML and Statistics Input Data Data PreProcessing Data integration Normalization Feature selection Dimension reduction Data Mining Pattern discovery Association & correlation Classification Clustering Outlier analysis ………… PostProcessing Pattern evaluation Pattern selection Pattern interpretation Pattern visualization This is a view from typical machine learning and statistics communities 17
- 18. Data Mining: OnWhat Kinds of Data? 18 Database-oriented data sets and applications Relational database, data warehouse, transactional database Advanced data sets and advanced applications Data streams and sensor data Time-series data, temporal data, sequence data (incl. bio-sequences) Structure data, graphs, social networks and multi-linked data Object-relational databases Heterogeneous databases and legacy databases Spatial data Multimedia database Text databases The World-Wide Web
- 19. RDBMS 19 A database thathas a collection of tables of data items, all of which is formally described and organized according to the relational model. Data in a single table represents a relation. Each table schema must identify a column or group of columns, called the p rim a ry ke y , to uniquely identify each row. A relationship can then be established between each row in the table and a row in another table by creating a fo re ig n ke y , a column or group of columns in one table that points to the primary key of another table.
- 20. RDBMS 20 • • • • • Database normalization: Therelational model offers various levels of refinement of table organization and reorganization . DBMS of a relational database is called an RDBMS, and is the software of a relational database. The relational database was first defined in June 1970 by Edgar Codd, of IBM's San Jose Research Laboratory. Codd's view of what qualifies as an RDBMS is summarized in Codd's 12 rules. A relational database has become the predominant choice in storing data.
- 21. 21 Relational database terminology. A relationis defined as a set of tuples that have the same attributes
- 22. RDMS(contd..) 22 Example :Allelectronics(Company describedby relation tables:Customer,item,employee and branch) Relation : customer is a group of entities describing the customer information(Cust_id,cust_name, Age,Occupation,annual income, credit information and category) Tables: used to represent the relationship between or among multiple entities Database queries(SQL): For data accessing using relational operations such as join, selection and projection
- 23. Mining Relational databases 23 Cango further by searching for trends or data patterns Examples Analyze customer data to predict the risk of customers based on their income ,age Detect deviations: sales comparison with previous year RDBMS are one of the most commonly available and richest information repositories for data mining
- 24. What is aData Warehouse? 24 Defined in many different ways, but not rigorously. A decision support database that is maintained separately from the organization’s operational database Support information processing by providing a solid platform of consolidated, historical data for analysis. “A data warehouse is a subject-oriented, integrated, time-variant, and nonvolatile collection of data in support of management’s decisionmaking process.”—W. H. Inmon Data warehousing: The process of constructing and using data warehouses
- 25. DATA WAREHOUSES 25 Is arepository of information collected from multiple sources, stored under a unified schema. Constructed via Data cleaning Data integration Data transformation Data Loading and periodic data refreshing
- 26.
- 27. DATA WAREHOUSES(contd…) 27 Data warehouseis modeled by a multidimensional data structure Data cube: precomputation &fast access of summarized data Each dimension corresponds to an attribute or a set of attributes in a schema Each cell stores the value of some aggregate measure (count, sum etc) Example: In Allelectronics the cube has three dimension : • Address(with city values, U S A, Canada, Mexico) • Time (with quarter values Q1,Q2,Q3,Q4) • Item(with type values )
- 28. Multidimensional Data 28 Sales volumeas a function of product, month, and region Re g io n Dimensions: Product, Location, Time Hierarchical summarization paths Industry Region Year Category Country Quarter Product Product City Office Month Month Day Week
- 29. A Sample DataCube 29 Pr TV PC VCR sum 1Qtr 2Qtr 3Qtr 4Qtr sum Total annual sales of TVs in U.S.A. U.S.A Canada Mexico sum Country od uc t Date
- 30. Data mining functionalities 30 Taskscan be classified : Predictive(makes prediction about values of data using known results found from different data) Descriptive( characterize properties of a target data set) Explore the properties of the data examined Data mining functionalities are used to specify the kinds of patterns Characterization and Discrimination The mining of frequent patterns, associations and correlations Classification and regression Cluster analysis Outlier analysis
- 31. Characterization and Discrimination 31 Datacharacterization is a summarization of the general characteristics or features of a target class of data Output of characterization can be presented in various forms Pie charts Bar charts Curves multidimensional data cube Multidimensional tables Descriptions presented in generalized relations- Characteristic rules Example: In Allelectronics : Sum m a riz e the c ha ra c te ris tic o f c us to m e rs who s p e nd m o re tha n $ 5 0 0 0 a y e a r a t A le c tro nic s lle this can be view in any dimension, such as on occupation to view these customers according to their type of employment.
- 32. Data Discrimination 32 Data discriminationis a comparison of the general features of the target class data objects against the general features of objects from one or more multiple contrasting class Output representation similar to characterization description Discrimination description expressed in the form of rules –Discrimination rules Target and contrasting class specified by the user Example: Us e r wa nt to c o m p a re the g e ne ra l fe a ture s o f s o ftwa re p ro d uc ts with s a le s tha t inc re a s e d by 1 0 % a nd d e c re a s e d by 3 0 % d uring the s a m e p e rio d
- 33. Mining Frequent Patterns,Associations, Correlations 33 Frequent pattern Frequent item sets(Milk, bread) Frequent subsequences(Latop ,digital camera ,memory card) Frequent sub structures (graphs ,trees) Mining frequent patterns leads to the discovery of interesting associations and correlation within data.
- 34. Association analysis(example) 34 Item frequentlypurchased together buys(X, ”computer”) =>buys(X, ”software”) [support=1%, confidence=50%] X - a variable representing a customer A confidence or certainty – 50%(chance) 1%(under analysis) Association rule- with single-dimension association rules “computer => software[1%,50%]”. Age(X,”20..29”) ^ income(X,”40K..49K”)=>buys(X ,”laptop”) [support=2%, confidence=60%] (Multidimensional association rule)
- 35. Classification and Regressionfor Predictive Analysis 35 Classification: the process of finding a model(function)that describes and distinguishes data classes or concepts Model derived from analysis of a set of training data Models are represented as Classification rules(IF-THEN rules) Decision trees Mathematical formulae or Neural networks Regression: Statistical methodology for numeric prediction
- 36. 36 Cluster Analysis andOutlier Analysis Cluster Analysis: Determining similarity among data on predefined attributes The most similar data are grouped into clusters Outlier Analysis Outliers: The dataset contain objects that do not required for the model of the data Analysis of outlier data is referred to as Outlier Analysis or Anomaly mining Detected using statstical tests
- 37. Which Technologies AreUsed? Machine Learning Applications Algorithm Pattern Recognition Statistics Visualization Data Mining Database Technology High-Performance Computing 37
- 38. Potential Applications ofData Mining Where there are data there are data mining applications 38 Data analysis and decision support ( Business Intelligence) Market analysis and management Risk analysis and management Target marketing, customer relationship management (CRM), market basket analysis, cross selling, market segmentation Forecasting, customer retention, improved underwriting, quality control, competitive analysis Fraud detection and detection of unusual patterns (outliers) Other Applications Text mining (news group, email, documents) and Web mining Stream data mining Bioinformatics and bio-data analysis
- 39. Major Issues inData Mining (1) Mining Methodology Mining knowledge in multi-dimensional space Data mining: An interdisciplinary effort Boosting the power of discovery in a networked environment Handling noise, uncertainty, and incompleteness of data Mining various and new kinds of knowledge Pattern evaluation and pattern- or constraint-guided mining User Interaction Interactive mining Incorporation of background knowledge Presentation and visualization of data mining results 39
- 40. Major Issues inData Mining (2) Efficiency and Scalability Efficiency and scalability of data mining algorithms Parallel, distributed, stream, and incremental mining methods Diversity of data types Handling complex types of data Mining dynamic, networked, and global data repositories Data mining and society Social impacts of data mining Privacy-preserving data mining Invisible data mining 40