3161607 – Big Data Analytics
WEB BASED DATA
MANAGEMENT OF APACHE HIVE
Submitted To : Prof. Pooja Bhatt
Present By :
Krupa Patel(190633116001)
Maitri Patel(190633116003)
Riya Soni(190633116004)
1
Outlines :
Origin
What is hive?
How Hive works?
Hive Architecture
Working of hive
Execution of Hive
limitations
Apache Hive Vs Pig
Hive Table
Summary
2
Origin:
Hive was Initially developed by Facebook.
Data was stored in Oracle database every night
ETL(Extract,Transform,Load) was performed on Data
The Data growth was exponential
– By 2006 1TB /Day
– By 2010 10 TB /Day
– By 2013 about 5000,000,000 per day..etc and
there was a need to find some way to manage the data
“effectively”.
3
What is Hive?
Hive is a Data warehouse infrastructure built on top of
Hadoop that can compile SQL Quires as Map Reduce
jobs and run the jobs in the cluster.
Suitable for semi and structured databases.
Capable to deal with different storage and file formats.
Provides HQL(SQL like Query Language) What Hive
is not.
Does not use complex indexes so do not response in
seconds.
But it scales very well , it works with data of peta byte
order.
4 It is not independent and its performance is tied
How Hive Works?
Hive Built on top of Hadoop – Think HDFS and Map
Reduce
Hive stored data in the HDFS
Hive compile SQL Quires into Map Reduce jobs and
run the jobs in the Hadoop cluster.
It stores schema in a database and processed data into
HDFS.
It is designed for OLAP.
We need reports to make operations better not to
conduct and operations.
5
We use ETL to populate data in DW
Hive Architecture
6
Hive Architecture
User Interface – Hive is a data warehouse infrastructure
software that can create interaction between user and HDFS.
Meta Store – Hive chooses respective database servers to
store the schema or Metadata of tables, databases, columns in
a table, their data types, and HDFS mapping.
HiveQL Process Engine – HiveQL is similar to SQL for
querying on schema info on the Metastore. It is one of the
replacements of traditional approach for MapReduce
program.
7
Hive Architecture
Execution Engine : The conjunction part of HiveQL
process Engine and MapReduce is Hive Execution
Engine. It uses the flavor of MapReduce.
HDFS or HBASE – Hadoop distributed file system or
HBASE are the data storage techniques to store data
into file system. Extreme scalability (up to 100 PB) –
Self-healing storage .
8
Working of Hive :
9
Execution of Hive :
Execute Query : The Hive interface such as
Command Line or Web UI sends query to Driver (any
database driver such as JDBC, ODBC, etc.) to execute.
Get Plan : The driver takes the help of query
compiler that parses the query to check the syntax and
query plan or the requirement of query.
Get Metadata : The compiler sends metadata request
to Meta store (any database).
Send Metadata: Meta store sends metadata as a
response to the compiler.
10
Execution of Hive :
Send Plan : The compiler checks the requirement and
resends the plan to the driver. Up to here, the parsing
and compiling of a query is complete.
Execute Plan: The driver sends the execute plan to
the execution engine.
Execute Job: The execution engine sends the job to
JobTracker, which is in Name node and it assigns this
job to TaskTracker, which is in Data node. Here, the
query executes MapReduce job.
11
Execution of Hive :
Metadata Ops : Meanwhile in execution, the
execution engine can execute metadata operations with
Meta store.
Fetch Result : The execution engine receives the
results from Data nodes.
Send Results : The execution engine sends those
resultant values to the driver. The driver sends the
results to Hive Interfaces.
12
Limitations:
The biggest limitation of Hadoop is that one have to
use M/R model (Map-Reduce Model). Other
limitations are as stated below:
* Not Reusable
* Error prone
* Multiple stage of Map/Reduce functions for complex
jobs.
*It’s just like asking a developer to write physical
execution plan in the DB.
13
Apache Hive vs. Apache Pig
14
Hive Table:
A Hive Table: -
Data: file or group of files in HDFS .
Schema: in the form of metadata stored in a relational
database
You have to define a schema if you have existing data in
HDFS that you want to use in Hive.
Schema and Data are separate.
15
Defining a Table
16
Managing Table
17
Loading Data
Use LOAD DATA to import data into Hive Table.
Use the word OVERWRITE to write over a file of the same
name
18
Insert Data
Use INSERT statement to populate data into a table from
another Hive table.
Overwrite is used to replace the data in the table, Otherwise
the data is appended to the table
19
Performing Queries (HiveQL):
SELECT
20
Summary
21
Thank You…!
22
