Apache Hadoop and its role in Big Data architecture - Himanshu Bari

© Hortonworks Inc. 2013
The Rise of Apache Hadoop…
…and its Role in Enterprise Data Architectures
Himanshu Bari
Sr. Product Manager, Hortonworks
Page 1

Topics
Page 2
Market trends &
emergence of
Hadoop
Hadoop’s role
and future
direction in the
Enterprise
Enterprise
Hadoop use
cases

Big Data & Big Impact
Page 3
Big Data
15x
growth rate of
machine
generated
data by 2020
Source: IDC
Big Impact
20%
Percentage by which
companies leveraging data
will outperform their peers
1.5M Data Savvy managers
needed
Source: Mckinsey

2013: CIOs take note…
Page 4
2013 STATE OF THE CIO SURVEY (Jan 2013)
When it comes to adoption of Big Data:
•  34% of IT executives surveyed classify their organization as late majority
•  25% of IT executives surveyed classify their organization as laggards

What is Apache Hadoop?
Page 5
D D
.... D
C C C
Petabyte scale reliable storage
management (HDFS) on commodity disks
Highly distributed computation framework
( MapReduce)
Apache Hadoop = Open Source Data Management Software

Quick History: Hadoop at Yahoo!
Source: http://developer.yahoo.com/blogs/ydn/posts/2013/02/hadoop-at-yahoo-more-than-ever-before/
Page 6

Topics
Page 7
Market Trends
& Emergence of
Hadoop
Hadoop’s role
and future
direction in the
Enterprise
Enterprise
Hadoop Use
Cases

Current Data ArchitectureAPPLICATIONS
DATA
SYSTEMS

TRADITIONAL
REPOS

RDBMS
EDW
MPP

DATA
SOURCES

OLTP,
POS

SYSTEMS

OPERATIONAL

TOOLS

MANAGE
&

MONITOR

Tradi:onal
Sources

(RDBMS,
OLTP,
OLAP)

DEV
&
DATA

TOOLS

BUILD
&

TEST

Business

Analy:cs

Custom

Applica:ons

Packaged

Applica:ons

Page 8
ETL/ELT

Current Data Architecture PressuredAPPLICATIONS
DATA
SYSTEMS

TRADITIONAL
REPOS

RDBMS
EDW
MPP

DATA
SOURCES

OLTP,
POS

SYSTEMS

OPERATIONAL

TOOLS

MANAGE
&

MONITOR

Tradi:onal
Sources

(RDBMS,
OLTP,
OLAP)

New
Sources

85% data growth

(sen:ment,
clickstream,
geo,
sensor,
…)

DEV
&
DATA

TOOLS

BUILD
&

TEST

Business

Analy:cs

Custom

Applica:ons

Packaged

Applica:ons

Page 9
ETL/ELT

Next generation data architectureAPPLICATIONS
DATA
SYSTEMS

TRADITIONAL
REPOS

RDBMS
EDW
MPP

DATA
SOURCES

OLTP,
POS

SYSTEMS

OPERATIONAL

TOOLS

MANAGE
&

MONITOR

Tradi:onal
Sources

(RDBMS,
OLTP,
OLAP)

New
Sources

85% data growth

(sen:ment,
clickstream,
geo,
sensor,
…)

DEV
&
DATA

TOOLS

BUILD
&

TEST

Business

Analy:cs

Custom

Applica:ons

Packaged

Applica:ons

ENTERPRISE

HADOOP

PLATFORM

Page 10
ETL/ELT

New architecture enables schema on read
Page 11
OLD WAY HADOOP WAY
Define table
with
Schema
Load only
table
conforming
data
CHANGE
?
Fight for eternity
Load
COMPLETE
data in
Hadoop
Read data
as you like
CHANGE
?
Just read differently

OS/VM
Cloud
Appliance

ENTERPRISE
HADOOP
PLATFORM

Evolution of Enterprise Hadoop
Page 12
HADOOP

CORE

PLATFORM

SERVICES

DATA

SERVICES

S:nger

HIVE
&

HCATALOG

PIG
HBASE

SQOOP

FLUME

NFS

WebHDFS

HDFS

MAP

REDUCE

YARN

TEZ
OTHER

OPERATIONAL

SERVICES

OOZIE

AMBARI

FALCON

Enterprise Readiness
High Availability, Disaster
Recovery, Rolling Upgrades,
Security and Snapshots
KNOX

OpenStack

YARN: General purpose resource
management framework
•  Why is it needed?
–  New ways of data processing graph and
stream processing have different resource
management needs than mapreduce
–  Need to improve scalability & utilization of
the clusters
–  Support multiple versions of mapreduce
•  How does it work?
–  Splits JobTracker responsibilities into a
global resource manager and a per-
application ApplicationMaster
–  Provides an extendible framework HDFS

MapReduce

Redundant, Reliable Storage
YARN:
Cluster
Resource
Management

Tez

Stream
Processing

Other

…
Page 13
HADOOP
CORE

Apache Tez (“Speed”): Alternative to
MapReduce
• Why is it needed?
– Widens the platform for Hadoop use cases beyond batch
– Crucial to improving the performance of low-latency applications
• Core idea-
– Create a pool of pre-allocated containers
–  Reuse containers for multiple tasks
Page 14
pluggable
input
Pluggable
Processor
Task
Pluggable
Output
HADOOP
CORE

Stinger: Improve Hive performance
and SQL compliance
Page 15
Improves existing
tools & preserves
investments
Enable Hive to
support interactive
workloads
Stinger Project
Simple Focus
Query
Planner
Hive
Execution
Engine
Tez
= 100X+
New File
Format
ORC file
= SQL Compliance
+
Data
Types
Windowing
&
Subqueries
+
DATA
SERVICES

Falcon: One-stop Shop for Data
Lifecycle management(DLM)
Data Management Needs Tools
Data Processing Oozie
Replication Sqoop
Retention Distcp
Scheduling Flume
Reprocessing Map / Reduce
Multi Cluster Management Hive and Pig Jobs
Falcon provides a single interface to orchestrate data lifecycle.
Sophisticated DLM easily added to Hadoop applications.
OPERATIONAL
SERVICES
Apache Falcon
Provides Orchestrates

Knox: Make Hadoop Security Simple
Simplify Security Aggregate Access Client Agility
Simplify security for both users
and operators.
Deliver unified and centralized
access to the Hadoop cluster to
give a ‘single application’ feel
Ensure service users are
abstracted from where services
are located and how services
are configured & scaled
PLATFORM
SERVICES
Hadoop Cluster
Authentication
&
Verification
Client
User Store
KDC, AD, LDAP
{REST}!
Knox
gateway
cluster

•  OpenStack provides operational agility and deployment choice
•  Hadoop is a net new workload and a perfect app for OpenStack
•  Integration marries two of the Largest Open Source Movements
–  Community-driven innovation outpaces any single vendor
–  Both are attracting major ecosystem players: IBM, RHT, HP, RAX, etc…
Page 18
Project Savanna to enable Hadoop
on OpenStack
CLOUD
PLATFORM
ENABLEMENT
Project Savanna
Automate deployment of
Apache Hadoop on
OpenStack

Topics
Page 19
Market Trends
& Emergence of
Hadoop
Hadoop’s role
and future
direction in the
Enterprise
Enterprise
Hadoop Use
Cases

Fundamental business drivers the same…
• Better
– Automation
– Transparency
– Segmentation
– Innovation & experimentation
• Faster
– Everything
• Cheaper
– Across the value chain
Page 20

6 Common TYPES OF DATA
1.  Sentiment
Understand how your customers feel about your brand and
products – right now
2.  Clickstream
Capture and analyze website visitors’ data trails and
optimize your website
3.  Sensor/Machine
Discover patterns in data streaming automatically from
remote sensors and machines
4.  Geographic
Analyze location-based data to manage operations where
they occur
5.  Server Logs
Research logs to diagnose process failures and prevent
security breaches
6.  Text
Understand patterns in text across millions of web pages,
emails, and documents
Value
Page 21

Financial services
• Industry specific drivers for Hadoop
– Increasing compliance regulatory pressure
– Bad guys never stop
– Never ending Macroeconomic volatility
– Cost pressures – more than ever
– Extreme competition
• Common use cases
– Fraud & risk reduction( eg. During new account creation)
– Sentiment based trading strategies
– Improve insurance underwriting based on usage and longer
history
– Data reservoir ( for archival, compliance inquiries etc.)
Page 22

Retail
– Increasingly ‘value sensitive’ and SMART consumer
– Constant margin pressure
– Emergence of the multi-channel approach
– 360 degree customer view ( behavior, location, sentiment etc.)
– Micro and dynamic segmentation
– Optimizations – Price, assortment, layout, supply chain
– Seasonal predictions – product styles, labor needs etc.
Page 23

Telcos
– Infrastructure under stress with rise of smart devices
– 4G/LTE investment needs CapEx but revenue growth largely flat
– Cloud computing changing the game
– Increasing competition with non Telcos
– Sitting on a gold mine of data
– Understanding customer behavior AND context (eg. location
based) in real-time
– Packaging and selling data
– Call Detail Record (CDR) & extended data record (XDR) analysis
for service quality improvement & capacity planning/optimization
– Customer churn analysis & prevention
Page 24

Healthcare & Pharma
– Sudden data deluge
– Health data initiative (HDI) by US Govt.
– Digitization of health records and rise of sensor data
– Huge accumulation of R&D data
– Rising healthcare costs
– Payors shift to outcome based payment models with providers as
well as pharmaceutical companies
– Improved patient outcome tracking
– Optimized patient recruitment for drug trials
– Reduce drug modeling time
– Improve insurance claim validation accuracy
Page 25

Manufacturing
– Proliferation of sensors
– Globalization of the supply chain
– Ongoing miniaturization of products
– Failure analysis to perform proactive maintenance
– Improving equipment quality by more frequent sample testing and
rigorous prototype testing
– Supply chain optimization
Page 26

Hadoop Summit
•  June 26-27, 2013- San Jose Convention Cntr
•  Co-hosted by Hortonworks & Yahoo!
•  Theme: Enabling the Next Generation
Enterprise Data Platform
•  90+ Sessions and 7 Tracks:
•  Community Focused Event
–  Sessions selected by a Conference Committee
–  Community Choice allowed public to vote for
sessions they want to see
•  Training classes offered pre event
–  Apache Hadoop Essentials: A Technical
Understanding for Business Users
–  Understanding Microsoft HDInsight and Apache
Hadoop
–  Developing Solutions with Apache Hadoop –
HDFS and MapReduce
–  Applying Data Science using Apache Hadoop
Page 27
hadoopsummit.org

Thank You
Follow us: @hortonworks
Page 28
http://hortonworks.com/products/hortonworks-sandbox/

Similar solution architecture across use
cases
Page
Hadoop
LOAD
SQOOP
FLUME
Web
HDFS
NFS
USE
DB
EDW
MPP
SOURCE
DATA
1
2
3
4
5
BATCH
STREAMING
STORM
Map
Reduce
PIG
INTERACTIVE
HIVE/SQL
ONLINE
HBASE
AMBARI
HCATALOG (table metadata)
PIG(data
processing)
HIVE
(data processing)
compute
&
storage
. . .
. . .
. .
compute
&
storage
.
.
YARN

Apache Hadoop and its role in Big Data architecture - Himanshu Bari

More Related Content

What's hot

Similar to Apache Hadoop and its role in Big Data architecture - Himanshu Bari

Recently uploaded

Apache Hadoop and its role in Big Data architecture - Himanshu Bari