BIG DATA
“We swim in a sea of data … and the sea level is rising rapidly.”
Selvarani Mylsamy
1/25/2016
1
Agenda
Why Big Data
What is Big Data
Classification of BigData
Who’s generating Big Data
Characteristics of Big Data
OLTP Vs OLAP
What’s driving Big Data
Predictive Analytics
Big Data Architecture
Tools used in Big Data
Applications of Big Data
1/25/2016
2
Why Big Data?
1/25/2016
3
What is Big Data?
1/25/2016
4
Definition of Big Data
1/25/2016
5
Classification of Big Data
1/25/2016
6
Example of Big Data in Airline Domain
1/25/2016
7
Model has changed
The Model of Generating/Consuming Data has Changed
Old Model: Few companies are generating data, all others are consuming data
New Model: All of us are generating data, and all of us are consuming data
1/25/2016
8
Who’s Generating Big Data?
Scientific instruments Mobile devices
(collecting all sorts of data) (tracking all objects all the time)
Social media and networks
(all of us are generating data)
Sensor technology and networks
(measuring all kinds of data)
The ability to manage, analyze, summarize, visualize, and discover knowledge from the collected
data in a timely manner and in a scalable fashion
1/25/2016
9
Who’s Generating Big Data?
1/25/2016
10
Explosion in Quantity of Data
1/25/2016
11
Characteristics of Big Data
1/25/2016
12
Volume
1-Scale (Volume)
Data Volume
44x increase from 2009 2020
From 0.8 zettabytes to 35zb
Data volume is increasing exponentially
1/25/2016
13
Variety
2-Complexity (Variety)
Various formats, types, and structures
Text, numerical, images, audio, video,
sequences, time series, social media data,
multi-dim arrays, etc…
Static data vs. streaming data
A single application can be
generating/collecting many types of data
To extract knowledge all these types
of data need to linked together
1/25/2016
14
Velocity
3-Speed (Velocity)
Data is being generated fast and need to be processed fast
Online Data Analytics
Late decisions missing opportunities
Examples
E-Promotions:
Based on your current location, your purchase history, what you like
send promotions right now for store next to you
Healthcare monitoring:
sensors monitoring your activities and body
any abnormal measurements require immediate reaction
1/25/2016
15
3Vs
1/25/2016
16
3Vs
Harnessing Big Data
OLTP: Online Transaction Processing (DBMSs)
OLAP: Online Analytical Processing (Data Warehousing)
RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)
1/25/2016
17
OLTP Vs OLAP
OLTP OLAP
Data Source Operational: ERP, CRM, legacy Management Information
apps, ... System, Decision Support
System
Typical users Staff Managers, Executives
Horizon Weeks, Months Years
Refresh Immediate Periodic
Data model Entity-relationship Multi-dimensional
Schema Normalized Star
Focus Update Data Retrieval (Reporting Data)
Space Small Large
Queries Simple Complex
1/25/2016
18
OLAP Overview
1/25/2016
19
What’s driving Big Data
- Optimizations and predictive analytics
- Complex statistical analysis
- All types of data, and many sources
- Very large datasets
- More of a real-time
- Ad-hoc querying and reporting
- Data mining techniques
- Structured data, typical sources
- Small to mid-size datasets
1/25/2016
20
What’s Predictive Analytics?
Data analysis with mathematical techniques from statistics, data
mining, and machine learning.
Used to uncover hidden patterns that yields competitive advantage
1/25/2016
21
Common customer scenarios for predictive analytics
1/25/2016
22
Advantages of Big Data
Drive incremental revenue
• Predict customer behavior across all channels
• Understand and monetize customer behavior
Improve operational effectiveness
• Machines/sensors: predict failures, network attacks
• Financial risk management: reduce fraud, increase security
Reduce data warehouse cost
• Integrate new data sources without increased database cost
• Provide online access to ‘dark data’
1/25/2016
23
Big Data Architecture
1/25/2016
24
Big Data Features
1/25/2016
25
What is Hadoop?
• Open source framework by Doug cutting in 2006
• Maintained by Apache Software foundation
• Distributed under Apache license
• Linux based set of tools
• Designed to store and process huge volumes of data efficiently
Core Components
1/25/2016
26
Hadoop cluster
1/25/2016
27
Example : Word Count
Count the occurrences of each word in a data set
1/25/2016
28
Example Contd
1/25/2016
29
Applications of Big Data
1/25/2016
30
References
http://web.cs.wpi.edu/~cs525/s13-MYE/
www.itskillsindemand.com
www.brandwatch.com
www.quora.com/Data-Warehousing/
https://svforum.org/.../Big%20Data%20for%20BI%20-%20Beyond%20t
http://www.docme.ru/doc/833138/
https://gigaom.com/
http://cdn.blog.profoundis.com/
1/25/2016
31
1/25/2016
32
1/25/2016
33
