Introduction to Hortonworks
Data Platform (HDP)
© Copyright IBM Corporation 2021
Course materials may not be reproduced in whole or in part without the prior written permission of IBM.
Unit objectives
• Describe the functions and features of HDP.
• List the IBM added value components.
• Describe the purpose and benefits of each added value component.
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Hortonworks Data Platform
overview
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Topics
• Hortonworks Data Platform overview
• Data flow
• Data access
• Data lifecycle and governance
• Security
• Operations
• Tools
• IBM added value components
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Hortonworks Data Platform
• HDP is a platform for data at rest.
• It is a secure, enterprise-ready open-source Apache Hadoop distribution
that is based on a centralized architecture (YARN).
• HDP has the following attributes:
▪ Open
▪ Central
▪ Interoperable
▪ Enterprise-ready
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Hortonworks Data Platform
Governance
Tools Security Operations
Integration
Data Lifecycle Zeppelin Ambari User Views Ranger Ambari
and Governance
Knox Cloudbreak
Falcon
Atlas ZooKeeper
Atlas
HDFS
Oozie
Encrpytion
Data workflow
Data Access
Sqoop
Batch Script SQL NoSQL Stream Search In-Mem Others
Flume HBase HAWQ
Map
Pig Hive Accumulo Storm Solr Spark Partners
Reduce
Kafka Phoenix Db2 Big SQL
Tez Tez Slider S T
NFS
YARN: Data Operating System
WebHDFS Hadoop Distributed File System (HDFS)
Data Management
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Data flow
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Topics
• Hortonworks Data Platform overview
• Data flow
• Data access
• Data lifecycle and governance
• Security
• Operations
• Tools
• IBM added value components
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Data Flow
Governance
Tools Security Operations
Integration
Data Lifecycle Zeppelin Ambari User Views Ranger Ambari
and Governance
Knox Cloudbreak
Falcon
Atlas ZooKeeper
Atlas
HDFS
Oozie
Encrpytion
Data workflow
Data Access
Sqoop
Batch Script SQL NoSQL Stream Search In-Mem Others
Flume HBase HAWQ
Map
Pig Hive Accumulo Storm Solr Spark Partners
Reduce
Kafka Phoenix Db2 Big SQL
Tez Tez Slider S T
NFS
YARN: Data Operating System
WebHDFS Hadoop Distributed File System (HDFS)
Data Management
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Kafka
• Apache Kafka is a fast, scalable, durable, and fault-tolerant publish-
subscribe messaging system.
▪ Used for building real-time data pipelines and streaming apps
• Often used in place of traditional message brokers like JMS and AMQP
because of its higher throughput, reliability and replication.
• Kafka works in combination with variety of Hadoop tools:
▪ Apache Storm
▪ Apache HBase
▪ Apache Spark
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Sqoop
• Tool to easily import information from structured databases (Db2,
MySQL, Netezza, Oracle, and mode.) and related Hadoop systems
(such as Hive and HBase) into your Hadoop cluster
• Can also be used to extract data from Hadoop and export it to relational
databases and enterprise data warehouses
• Helps offload some tasks such as ETL from Enterprise Data Warehouse
to Hadoop for lower cost and efficient execution
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Data access
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Topics
• Hortonworks Data Platform overview
• Data flow
• Data access
• Data lifecycle and governance
• Security
• Operations
• Tools
• IBM added value components
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Data access
Governance
Tools Security Operations
Integration
Data Lifecycle Zeppelin Ambari User Views Ranger Ambari
and Governance
Knox Cloudbreak
Falcon
Atlas ZooKeeper
Atlas
HDFS
Oozie
Encrpytion
Data workflow
Data Access
Sqoop
Batch Script SQL NoSQL Stream Search In-Mem Others
Flume HBase HAWQ
Map
Pig Hive Accumulo Storm Solr Spark Partners
Reduce
Kafka Phoenix Db2 Big SQL
Tez Tez Slider S T
NFS
YARN: Data Operating System
WebHDFS Hadoop Distributed File System (HDFS)
Data Management
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Hive
• Apache Hive is a data warehouse system built on top of Hadoop.
• Hive facilitates easy data summarization, ad-hoc queries, and the
analysis of very large datasets that are stored in Hadoop.
• Hive provides SQL on Hadoop
▪ Provides SQL interface, better known as HiveQL or HQL, which allows for
easy querying of data in Hadoop
• Includes HCatalog
▪ Global metadata management layer that exposes Hive table metadata to
other Hadoop applications.
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Pig
• Apache Pig is a platform for analyzing large data sets.
• Pig consists of a high-level language called Pig Latin, which was
designed to simplify MapReduce programming.
• Pig's infrastructure layer consists of a compiler that produces
sequences of MapReduce programs from this Pig Latin code that you
write.
• The system is able to optimize your code, and "translate" it into
MapReduce allowing you to focus on semantics rather than efficiency.
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
HBase
• Apache HBase is a distributed, scalable, big data store.
• Use Apache HBase when you need random, real-time read/write
access to your big data.
▪ The goal of the HBase project is to be able to handle very large tables of
data that are running on clusters of commodity hardware.
• HBase is modeled after Google's BigTable and provides BigTable-like
capabilities on top of Hadoop and HDFS. HBase is a NoSQL data store.
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Accumulo
• Apache Accumulo is a sorted, distributed key/value store that provides
robust, scalable data storage and retrieval.
• Based on Google’s BigTable and runs on YARN
▪ Think of it as a "highly secure HBase"
• Features:
▪ Server-side programming
▪ Designed to scale
▪ Cell-based access control
▪ Stable
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Spark
• Apache Spark is a fast and general engine for large-scale data
processing.
• Spark has a variety of advantages including:
▪ Speed
− Run programs faster than MapReduce in memory
▪ Easy to use
− Write apps quickly with Java, Scala, Python, R
▪ Generality
− Can combine SQL, streaming, and complex analytics
▪ Runs on a variety of environments and can access diverse data sources
− Hadoop, Mesos, standalone, cloud…
− HDFS, Cassandra, HBase, S3…
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Storm
• Apache Storm is an open source distributed real-time computation
system.
▪ Fast
▪ Scalable
▪ Fault-tolerant
• Used to process large volumes of high-velocity data
• Useful when milliseconds of latency matter and Spark isn't fast enough
▪ Has been benchmarked at over a million tuples processed per second per
node
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Data lifecycle and governance
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Topics
• Hortonworks Data Platform overview
• Data flow
• Data access
• Data lifecycle and governance
• Security
• Operations
• Tools
• IBM added value components
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Data Lifecycle and Governance
Governance
Tools Security Operations
Integration
Data Lifecycle Zeppelin Ambari User Views Ranger Ambari
and Governance
Knox Cloudbreak
Falcon
Atlas ZooKeeper
Atlas
HDFS
Oozie
Encrpytion
Data workflow
Data Access
Sqoop
Batch Script SQL NoSQL Stream Search In-Mem Others
Flume HBase HAWQ
Map
Pig Hive Accumulo Storm Solr Spark Partners
Reduce
Kafka Phoenix Db2 Big SQL
Tez Tez Slider S T
NFS
YARN: Data Operating System
WebHDFS Hadoop Distributed File System (HDFS)
Data Management
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Atlas
• Apache Atlas is a scalable and extensible set of core foundational
governance services
▪ Enables enterprises to effectively and efficiently meet their compliance
requirements within Hadoop
• Exchange metadata with other tools and processes within and outside
of Hadoop
▪ Allows integration with the whole enterprise data ecosystem
• Atlas Features:
▪ Data classification
▪ Centralized auditing
▪ Centralized lineage
▪ Security and policy engine
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Security
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Topics
• Hortonworks Data Platform overview
• Data flow
• Data access
• Data lifecycle and governance
• Security
• Operations
• Tools
• IBM added value components
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Security
Governance
Tools Security Operations
Integration
Data Lifecycle Zeppelin Ambari User Views Ranger Ambari
and Governance
Knox Cloudbreak
Falcon
Atlas ZooKeeper
Atlas
HDFS
Oozie
Encrpytion
Data workflow
Data Access
Sqoop
Batch Script SQL NoSQL Stream Search In-Mem Others
Flume HBase HAWQ
Map
Pig Hive Accumulo Storm Solr Spark Partners
Reduce
Kafka Phoenix Db2 Big SQL
Tez Tez Slider S T
NFS
YARN: Data Operating System
WebHDFS Hadoop Distributed File System (HDFS)
Data Management
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Ranger
• Centralized security framework to enable, monitor, and manage
comprehensive data security across the Hadoop platform
• Manage fine-grained access control over Hadoop data access
components like Apache Hive and Apache HBase
• Using Ranger console can manage policies for access to files, folders,
databases, tables, or column with ease
• Policies can be set for individual users or groups
▪ Policies enforced within Hadoop
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Operations
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Topics
• Hortonworks Data Platform overview
• Data flow
• Data access
• Data lifecycle and governance
• Security
• Operations
• Tools
• IBM added value components
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Operations
Governance
Tools Security Operations
Integration
Data Lifecycle Zeppelin Ambari User Views Ranger Ambari
and Governance
Knox Cloudbreak
Falcon
Atlas ZooKeeper
Atlas
HDFS
Oozie
Encrpytion
Data workflow
Data Access
Sqoop
Batch Script SQL NoSQL Stream Search In-Mem Others
Flume HBase HAWQ
Map
Pig Hive Accumulo Storm Solr Spark Partners
Reduce
Kafka Phoenix Db2 Big SQL
Tez Tez Slider S T
NFS
YARN: Data Operating System
WebHDFS Hadoop Distributed File System (HDFS)
Data Management
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Ambari
• For provisioning, managing, and monitoring Apache Hadoop clusters.
• Provides intuitive, easy-to-use Hadoop management web UI backed by
its RESTful APIs
• Ambari REST APIs
▪ Allow application developers and system integrators to easily integrate
Hadoop provisioning, management, and monitoring capabilities in their own
applications
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Cloudbreak
• A tool for provisioning and managing Apache Hadoop clusters in the
cloud
• Automates launching of elastic Hadoop clusters
• Policy-based autoscaling on several cloud infrastructure platforms,
including:
▪ Microsoft Azure
▪ Amazon Web Services
▪ Google Cloud Platform
▪ OpenStack
▪ Platforms that support Docker container
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
Oozie
• Oozie is a Java based workflow scheduler system to manage Apache
Hadoop jobs
• Oozie Workflow jobs are Directed Acyclical Graphs (DAGs) of actions
• Integrated with the Hadoop stack
▪ YARN is its architectural center
▪ Supports Hadoop jobs for MapReduce, Pig, Hive, and Sqoop
Introduction to Hortonworks Data Platform (HDP) © Copyright IBM Corporation 2021
