CIT650
Introduction to Big Data
Principles of Big Data
1
Today’s Agenda
Big Data Phenomena
Big Data 1.0 Systems
Big Data 2.0 Systems
2
Part I
Big Data Phenomena
3
Big Data
Data is key resource in the modern world.
According to IBM, we are currently creating 2.5 quintillion bytes of
data everyday.
IDC predicts that the world wide volume of data will reach 40 zettabytes
by 2020.
The radical expansion and integration of computation, networking, dig-
ital devices and data storage has provided a robust platform for the
explosion in big data.
4
Big Data
5
On the Verge of A Disruptive Century: Breakthroughs
Gene Ubiquitous
Sequencing and Computing
Biotechnology
Smaller, Faster,
Cheaper Sensors
Faster
Communication
6
Big Data Applications are Everywhere
7
Big Data: What Happens in the Internet in a Minute?
8
Data Generation and Consumption Model is Changing
9
Data Generation and Consumption Model is Changing
Old Model: Few companies (producers) are generating data, all
others are consuming data
New Model: All of us are generating data, and all of us are
consuming data
10
Big Data
11
Big Data
Data generation and consumption is becoming a main part of people’s
daily life especially with the pervasive availability and usage of Internet
technology and applications.
12
Your Smart Phone is now Very smart
13
Internet of Things (IoT)
A network devices, connect directly with each other to capture, share
and monitor vital data automatically through a SSL that connects a
central command and control server in the cloud
Enabling communication between devices, people & processes to ex-
change useful information & knowledge that create value for humans
A global Network Infrastructure linking Physical & Virtual Objects
Infrastructure: Internet and Network developments
Specific object identification, sensor, and connection capability
14
Big Data: Internet of Things
15
Prediction of IoT Usage1
1
https://www.ericsson.com/
16
Why IoT opportunity is growing now?
Affordable hardware: Costs of actuators & sensors have been cut in
half over last 10 years
Smaller, more powerful hardware: Form factors of hardware have
shrunk to millimeter or even nanometer levels
Ubiquitous & cheap mobility: Cost for mobile devices, bandwidth
and data processing has declined over last 10 years
Availability of supporting tools: Big data tools & cloud based in-
frastructure have become widely available
17
Smart X Phenomena
18
What it all produce?
Data ... Data ... Data
19
Big Data: Activity Data
Simple activities like listening to music or reading a book are now
generating data.
Digital music players and eBooks collect data on our activities.
Your smart phone collects data on how you use it and your web
browser collects information on what you are searching for.
Your credit card company collects data on where you shop and your
shop collects data on what you buy.
It is hard to imagine any activity that does not generate data.
20
Big Data
The cost of sequencing one human genome has fallen from $100
million in 2001 to $1K in 2015
21
New Types of Data
22
New Types of Data
23
The Data Structure Evolution Over the Years
24
What Means Big Data?
25
Big Data (3V)
26
Big Data (5V)
27
Big Data
28
Big Data Definition
McKinsey global report described big data as the next frontier for in-
novation and competition.
The report defined big data as ”Data whose scale, distribution, di-
versity, and/or timeliness require the use of new technical architectures
and analytics to enable insights that unlock the new sources of business
value”
29
Big Data Revolution
30
IBM 5MB Hard Disk ;-)
31
Recent Advances in Computational power
Cheaper, larger, and faster disk storage
You can now put all your large database on disk
Cheaper, larger, and faster memory
You may even be able to accommodate it all in memory
Cheaper, more capable, and faster processors
Parallel computing architectures:
Operate on large datasets in reasonable time
Try exhaustive searches and brute force solutions
32
Big Data
Moore’s Law: The information density on silicon integrated circuits
double every 18 to 24 months
Users expect more sophisticated information
33
Your Pocket Size Terabytes Hard Disk
34
Hardware Advancements Enable Big Data Processing
35
Scale Up VS Scale Out
36
The Data Overload Problem
37
The Data Overload Problem
Data is growing at a phenomenal rate. It has become massive, opera-
tional, and opportunistic
Drowning in data but starving for knowledge
The hidden information and knowledge in these mountains of data are
really the most useful
38
The Data Overload Problem
39
Fourth Paradigm
Jim Gray, a database pioneer, described the big data phenomena as
the Fourth Paradigm and called for a paradigm shift in the computing
architecture and large scale data processing mechanisms.
The first three paradigms were experimental, theoretical and, more
recently, computational science
40
Fourth Paradigm
41
Fourth Paradigm
Thousand years ago - Experimental Science
Description of natural phenomena
Last few hundreds years - Theoretical Science
Newton’s laws, Maxwell’s equation , ...
Last few decades - Computational Science
Simulations of complex phenomena
Today - Data-Intensive Science
Scientists overwhelmed with datasets from many different sources
Data captured by instruments
Data generated by simulations
Data generated by sensor networks
42
Computing Clusters
Many racks of computers, thousands of machines per cluster.
Limited bisection bandwidth between racks.
43
Data Centers
44
Big Data is a Competitive Advantage
45
Big Data is a Competitive Advantage
”It’s not who has the
best algorithm that
wins, It’s who has
the most data”
Andrew Ng
46
Data is the new Oil/Gold
47
Big Data Processing Systems
Big Data is the New Oil
and
Big Data Processing Systems is the Machinery
48
Part II
Big Data 1.0 System: The Hadoop
Decade
49
A Little History: Two Seminal contributions
”The Google File System”2
Describes a scalable, distributed, fault-tolerant file system tailored for
data-intensive applications, running on inexpensive commodity hardware,
delivers high aggregate performance
”MapReduce: Simplified Data Processing on Large Clusters”3
Describes a simple programming model and an implementation for pro-
cessing large data sets on computing clusters.
2
S. Ghemawat, H. Gobioff, S. Leung. The Google file system. SOSP 2003
3
J. Dean, S. Ghemawat. MapReduce: Simplified Data Processing on Large Clusters.
OSDI 2004
50
Hadoop4: A Star is Born
Hadoop is an open-source software
framework that supports data-intensive
distributed applications and clones the
Google’s MapReduce framework.
It is designed to process very large
amount of unstructured and complex
data.
It is designed to run on a large number
of machines that don’t share any memory
or disks.
It is designed to run on a cluster of ma-
chines which can put together in rela-
tively lower cost and easier maintenance.
4
http://hadoop.apache.org/
51
Key Aspects of Hadoop
52
Hadoop’s Success
Big Data 1.0 = Hadoop
53
Hadoop’s Success5
Big Data 1.0 = Hadoop
5
https://www.google.com/trends/
54
The Always Dilemma: Does One Size Fit All?!
55
Big Data 2.0 Processing Systems
Big Data 2.0 ! = Hadoop
Domain-specific, optimized and vertically focused systems
Cloudera Impala
Apache Samza
Apache S4
Trinity
Apache Flink PowerGraph Apache Tajo
Google MapReduce Apache Spark Apache Storm GraphLab Facebook Presto Apache Phoenix
Hadoop Apache Hive Google Pregel Apache Giraph IBM Big SQL GraphX Apache Tez
2004 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 2015
56
Big Graphs
Google estimates that the total number of
web pages exceeds 1 trillion; experimental
graphs of the World Wide Web contain more
than 20 billion nodes and 160 billion edges.
Facebook reportedly consists of more than a
billion users (nodes) and more than 140 bil-
lion friendship relationships (edges) in 2012.
The LinkedIn network contains almost 260
million nodes and billions of edges.
Linked data contains about 31 billion triples.
57
Hadoop for Big Graphs?!
Popular graph query/analysis operations
such as: Page rank, Pattern matching,
Shortest path, Clustering (e.g. Max clique,
triangle closure), Community detection,...,
etc. are iterative in nature.
MapReduce programming model does not
directly support iterative data analysis. Pro-
grammers may implement iterative programs
by manually issuing multiple MapReduce
jobs and orchestrating their execution using
a driver program which wastes I/O, network
bandwidth and CPU resources.
It is not intuitive to think of graphs as
key/value pairs or matrices.
58
Pregel/Giraph6
In 2010, Google introduced the Pregel sys-
tem as a scalable platform for implementing
graph algorithms.
Pregel relies on a vertex-centric approach
and is inspired by the Bulk Synchronous Par-
allel (BSP) model.
In 2012, Apache Giraph was launched as an
open source project that clones the concepts
of Pregel and leverages the Hadoop infras-
tructure.
Other Projects: Spark GraphX (Apache),
GoldenOrb (Apche), GraphLab (CMU) and
Signal/Collect (UZH).
6
https://giraph.apache.org/
59
Big Streaming Data
Every day, Twitter generates more than 12
TB of tweets.
New York Stock Exchange captures 1 TB of
trade information.
About 30 billion radio-frequency identifica-
tion (RFID) tags are created every day.
Hundreds of millions of GPS devices sold ev-
ery year.
60
Static Data Computation vs Streaming Data Computation
61
Hadoop for Big Streams?!
From the stream-processing point of view,
the main limitation of the original implemen-
tation of the MapReduce framework is that
it was designed so that the entire output of
each map and reduce task is materialized
into a local file before it can be consumed
by the next stage.
This materialization step enables the im-
plementation of a simple and elegant
checkpoint/restart fault-tolerance mecha-
nism. But it causes significant delay for jobs
with real-time processing requirements.
Some Hadoop-based Trials include: MapRe-
duce Online and Incoop.
62
Twitter Storm7
Open source project developed by Nathan
Marz and acquired by Twitter in 2012.
Storm is a distributed stream-processing
system with the following key design fea-
tures: horizontal scalability, guaranteed re-
liable communication between the process-
ing nodes, fault tolerance and programming-
language agnosticism.
A Storm cluster is superficially similar to a
Hadoop cluster. One key difference is that a
MapReduce job eventually finishes, whereas
a Storm job processes messages forever (or
until the user kills it).
Other Projects: Flink, Apex, Spark Stream-
ing and Kafka Streams.
7
http://storm.apache.org/
63
Massively Parallel Processing (MPP) Optimized SQL query
engines
Big Data Management 64 / 80
NoSQL Databases8
NoSQL database systems represent a new generation of low-cost, high-
performance database software which is increasingly gaining more and
more popularity.
These systems promise to simplify administration, be fault-tolerant and
able to scale on commodity hardware (Scale out).
8
http://nosql-database.org/
65
Data Storage Options
66
Big Data Landscape
67
Big Data Landscape
68
Big Data Market Size9
9
https://www.statista.com/
69
The End
70
