BIG DATA
CHAPTER-5
Introduction to HBase
Introduction
In the early days ,data used to be less and
structured.
Data could be easily stored in a relational
database.
After the evaluation of internet
huge amount of structured and
semi structured data got
generated.
Storing and processing the data
using RDBMS becomes a
problem
H Base
• Open source software.
• Non relational.
• Distributed column oriented
database.
• It run on top of HDFS.
Different from RDBMS
• Not a SQL database
• Not relational
• No joins
• No query language
• Not a drop in
replacement of
RDBMS.
Features
• Linear Scalability
• Automatic and configurable
shading of table
• Automatic failure support
• Strictly consistent read and
writes.
• Provide real time random read
write access to data stored in
hdfs.
• Attach nicely with Hadoop map
reduce.
• Easy java API for client access.
• Import of large amount of data.
• Backup option
• Bloom filter for real time
quarries.
Limitation of HBase
• It takes a very long time to recover if the H Master goes down. It takes
a long time to activate another node if the first nodes go down.
• In HBase, cross data operations and join operations are very difficult
to perform, even if we join operations by using Map Reduce, it
requires a lot of time to design and develop.
• HBase needs a new format when we want to migrate from RDBMS
external sources to HBase servers.
• It is very challenging in HBase to support querying process. We need
integration support from SQL layers like Apache Phoenix to write
queries to trigger the information from the database.
• It takes enormous time to develop security factor to grant access to
the users.
Pig
• Pig Latin: A not-so-foreign
language for data processing.
Introduction
• Apache pig work over map reduce.
• It is a tool/platform which is used to
analyze large data.
• We can perform all data manipulation
operation in Hadoop using Apache pig.
• Pig provide high level language known
as Pig Latin.
• Pig Latin help the user to develop their
own functions for reading, writing and
processing data.
Pig Architecture
PIG
Map Reduce
Parse statements
Pig Latin Compile
script
Map Reduce
Optimize
Plan
HDFS
Features of PIG
• Rich in operators: It provides a large number of operator operation.
• Ease of programming: As the pig Latin is similar to SQL.
• UDFs : Pig provide the facility to create user defined function.
• Handles all kinds of data : Apache pig can analyzes all kind of data
structured and unstructured.
Work flow of PIG.
• Programmers need to write script using Pig Latin Language.
• All scripts are internally converted to map reduce task.
• Apache pig contains a component as PIG ENGINE that accept Latin
script as input and convert those script to map reduce jobs.
Install PIG.
Install PIG.
Install PIG.
Install PIG.
Install PIG.
Install PIG.
Can Elephant fly?
Can Hadoop be
used more
efficiently?
Let See…
Ideas
Let have a red bull for wings…
Not a great
idea.
Shrink
After
Before
Genetic change
Before After
Behind the scenes…?
Facebook
initially
developed hive
What is hive?
• Hive is a data ware house infrastructure built on top of Hadoop.
• Support analysis of large dataset stored in Hadoop.
• Provide sql query language called HIVEQL.
• To provide quick query response ,it provide indexing.
Architecture of hive
Working of hive
Install HIVE
Java SE - Downloads | Oracle Technology Network | Oracle
Install HIVE
Install HIVE
Install HIVE
Install HIVE
Hadoop 2.4.1 Subversion https://svn.apache.org/repos/asf/hadoop/common -r
1529768
Compiled by hortonmu on 2013-10-07T06:28Z
Compiled with protoc 2.5.0
From source with checksum 79e53ce7994d1628b240f09af91e1af4
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hadoop
Install Hive
Install Hive
Install Hive
Install Hive
Install Hive
Install Hive
Install Hive
Install Hive
Install Hive
javax.jdo.PersistenceManagerFactoryClass =
org.jpox.PersistenceManagerFactoryImpl
org.jpox.autoCreateSchema = false
org.jpox.validateTables = false
org.jpox.validateColumns = false
org.jpox.validateConstraints = false
org.jpox.storeManagerType = rdbms
org.jpox.autoCreateSchema = true
org.jpox.autoStartMechanismMode = checked
org.jpox.transactionIsolation = read_committed
javax.jdo.option.DetachAllOnCommit = true
javax.jdo.option.NontransactionalRead = true
javax.jdo.option.ConnectionDriverName = org.apache.derby.jdbc.ClientDriver
javax.jdo.option.ConnectionURL = jdbc:derby://hadoop1:1527/metastore_db;create = true
javax.jdo.option.ConnectionUserName = APP
javax.jdo.option.ConnectionPassword = mine
Install Hive
Install Hive
Hive - Data Types
1. Column Types
2. Literals
3. Null Values
4. Complex Types
Column Types
Column type are used as column data types of Hive. They are as follows:
1. Integral Types
2. String Types
3. Timestamp
4. Dates
5. Decimals
6. Union Types
Literals
1.Floating Point Types
Floating point types are nothing but numbers with decimal points. Generally, this
type of data is composed of DOUBLE data type.
2.Decimal Type
Decimal type data is nothing but floating point value with higher range than
DOUBLE data type. The range of decimal type is approximately -10-308 to 10308.
Complex Types
Create Database
Drop Database
Create Table
Example
Syntax of example
Load Data Statement
Example
Alter Table Statement
Change Statement
Change Statement
Drop Table Statement
Example
Partition
Renaming Partition
Dropping a Partition
Operator
There are four types of operators in Hive:
1. Relational Operators
2. Arithmetic Operators
3. Logical Operators
4. Complex Operators
Built in Function
the built-in functions available in Hive. The functions look quite
similar to SQL functions, except for their usage.
Creating a View
Dropping a View
Creating a Index
Dropping a Index
Select Query
Order By
Group By
Join Table
Join
Left Outer Join
Right Outer Join
Full Outer Join
Physical Layout of hive
• Warehouse directory in hdfs.
• User/hive/warehouse
• Tables : subdirectories of warehouse.
• Partitions : subdirectories of
corresponding Table Directory.
Encapsulation
• HIVEQL queries is converted to map reduce code using hive engine.
• Hive engine translate all queries into a directed acyclic graph of map-
reduce jobs.
• These map reduce jobs are sent to Hadoop for execution.
Dependencies
• /user/hive directory is created as soon as the hive session is started first
time.
• /user/hive/warehouse directory shall be accessible by everyone.
• Hadoop dfs –chmod –R 1777/user/hive/warehouse.
• Recommended to activate sticky note if supported.
Hive Command line Interface
• HIVE CLI can be invoked by hike command.
• %hive
Hive SQL script
Hive QL
• HIVE QL is similar to SQL query language.
• DML(Data manipulation language)
• Select
• DDL(Data Definition Language)
• SHOW TABLE
• CREATE TABLE
• ALTER TABLE
• DROP TABLE
Play with hive
Loading delimited data
Normal Table VS External Table
Normal Table External Table
Normal table are created under External table read directly from hdfs file.
warehouse directory.
Normal table are directly visible through External table are not visible in
hdfs directory browsing. warehouse directory.
On dropping a normal table, the source Only dropping the external table only the
data and table metadata both are metadata is deleted.
deleted.
Joins
• HIVE QL supports join on only equality expressions. Complex Boolean
Expression inequality conditions are not supported.
• More than 2 table can be joined.
• Number of map-reduce jobs are generated for a join depend on the
columns being used.
• If same col is used for all the tables, then n=1
• Otherwise n>1.
Data type and format
Primitive data types
Numeric Date/Time String Miscellaneous
• TIME STAMP • STRING • BOOLEAN
• DATE • VARCHAR • BINARY
• INTERVAL • CHAR
Integral Floating
• TINYINT
• SMALL INT • FLOAT
• INTEGER • DOUBLE
• BIGINT
Data format in hive
TEXTFILE ORC
PARQUET
Apache hive data file format.
SEQUENCE
FILE
AVRO
RCFILE
PIG VS HIVE
PIG HIVE
Script language Sql like language
Comparatively less no of line than map Comparatively less no of lines than
reduce. map reduce and pig.
No partition Yes partition
Pig is mainly used for programming Hive mainly used for data analysts.
Pig support Avro Hive does not support Avro
www.paruluniversity.ac.in
