[Approved by AICTE, Govt. of India & Affiliated to Dr.
APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Program-6
Aim: Installation of Hive along with practice examples.
Theory: Apache Hive is a distributed, fault-tolerant data warehouse system that enables
analytics at a massive scale. A data warehouse provides a central store of
information that can easily be analyzed to make informed, data driven decisions.
Hive allows users to read, write, and manage petabytes of data using SQL.
Prerequisites:
JDK(Java) installed in the system
Hadoop installed
7 Zip
Steps:
Step 1: Check whether the Java is installed or not using the following command.
$ java -version
Step 2: Check whether the Hadoop is installed or not using the following command.
$ hadoop version
Step 3: Download the Apache Hive file apache-hive-2.3.9-bin.tar.gz using Hive download
directory.
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Step 4: Unzip and Install Hive:
The following command is used to verify the download and extract the hive archive:
$ tar zxvf apache-hive-2.3.9-bin.tar.gz
$ ls
On successful download, you get to see the following response:
apache-hive-2.3.9-bin apache-hive-2.3.9-bin.tar.gz
Step 5: Copying files to /user/local/hive directory:
We need to copy the files from the super user “su -”. The following commands are
used to copy the files from the extracted directory to the /user/local/hive, directory.
$ su -
password:
# cd /home/cloudera/user/Download
# mv apache-hive-2.3.9-bin /usr/local/hive
# exit
Step 6: Setting up environment for Hive:
You can set up the Hive environment by appending the following lines to ~/.bashrc file:
export HIVE_HOME=/usr/local/hive
export PATH=$PATH:$HIVE_HOME/bin
export CLASSPATH=$CLASSPATH:/usr/local/Hadoop/lib/*:.
export CLASSPATH=$CLASSPATH:/usr/local/hive/lib/*:.
The following command is used to execute ~/.bashrc file:
$ source ~/.bashrc
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Step 7: Configuring Hive:
To configure Hive with Hadoop, you need to edit the hive-env.sh file, which is placed
in the $HIVE_HOME/conf directory. The following commands redirect to Hive config folder
and copy the template file:
$ cd $HIVE_HOME/conf
$ cp hive-env.sh.template hive-env.sh
Edit the hive-env.sh file by appending the following line:
export HADOOP_HOME=/user/local/27adoop
Hive installation is completed successfully.
Step 8: Verify Hive Installation by following command:
$ bin/hive
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Program-7
Aim: Installation of HBase, Installing thrift along with Practice examples.
Theory: HBase provides low latency random read and write access to petabytes of data by
distributing requests from applications across a cluster of hosts. Each host has
access to data in HDFS and S3, and serves read and write requests in milliseconds.
Steps:
Step 1: Download the HBase file hbase-3.0.0-beta-1-bin.tar.gz from Apache website.
Step 2: Unzip and Move:
$ cd /home/cloudera/user/Downloads
$ sudo tar -zxvf hbase-3.0.0-beta-1-bin.tar.gz
$ sudo mv hbase-3.0.0-beta-1 /usr/local/hbase
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Step 3: Edit hbase-env.sh and hbase-site.xml:
$ cd /usr/local/hbase/conf
In the hbase-env.sh file, you need to export JAVA_HOME path. Here, you need to check the
JAVA_HOME path on your system. You can display your java home using the command
given below:
{} $ echo $JAVA_HOME
On this system the above command gives the output /usr/lib/jvm/java-7-openjdk-amd64
To update the hbase-env.sh, we need to run the command given below:
$ sudo nano hbase-env.sh
Copy and paste the below line in the hbase-env.sh file:
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export HBASE_REGIONSERVERS=/usr/local/hbase/conf/regionservers
export HBASE_MANAGES_ZK=true
Use Ctrl+X and Y to save.
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Your hbase-env.sh file will look like the image given below:
Now update the .bashrc file to export hbase variables:
$ sudo nano ~/.bashrc
Copy and paste the below lines at the end of .bashrc:
#HBASE VARIABLES START
export HBASE_HOME=/usr/local/hbase
export PATH=$PATH:$HBASE_HOME/bin
#HBASE VARIABLES END
Image below explains how to append the HBASE variables:
Use Ctrl+X and Y to save.
To make the above changes permanent in .bashrc, we need to run the command given
below:
$ source ~/.bashrc
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Now to update the hbase-site.xml file, use the command given below:
$ cd /usr/local/hbase/conf
$ sudo nano hbase-site.xml
Add the following lines between configuration tags:
<configuration>
<property>
<name>hbase.rootdir</name>
<value>hdfs://localhost:54310/hbase</value>
</property>
<property>
<name>hbase.zookeeper.property.dataDir</name>
<value>/usr/local/hadoop/zookeeper</value>
</property>
<property>
<name>hbase.cluster.distributed</name>
<value>true</value>
</property>
</configuration>
Use Ctrl+X and Y to save
Your hbase-site.xml file will look like the image given below:
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Step 4: Starting HBASE:
$ cd /usr/local/hbase/bin
$ sudo chown -R hduser:hadoop /usr/local/hbase/
$ ./start-hbase.sh
You will get the following message which looks as shown in the image given below at the
time of starting:
Once you need to run jps command to check the hbase daemons: {} $ jps
You will get the output as shown in the image given below:
To check the HBase Directory in HDFS:
$ hadoop fs -ls hdfs://localhost:54310/hbase/
You will get the output as shown in the image given below:
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Program-8
Aim: Write PIG Commands: Write Pig Latin scripts sort, group, join, project and filter your
data.
Pig Latin Script for Sorting:
The ORDER BY operator is used to display the contents of a relation in a sorted order based
on one or more fields.
Syntax:
grunt> Relation_name2 = ORDER Relatin_name1 BY (ASC|DESC);
Pig Latin Script for Group:
The GROUP operator is used to group the data in one or more relations. It collects the data
having the same key.
Syntax:
grunt> Group_data = GROUP Relation_name BY age;
Pig Latin Script for Join:
The JOIN operator is used to combine records from two or more relations. While performing
a join operation, we declare one (or a group of) tuple(s) from each relation, as keys. When
these keys match, the two particular tuples are matched, else the records are dropped.
Joins can be of the following types −
Self-join
Inner-join
Outer-join − le join, right join, and full join
Syntax:
Given below is the syntax of performing self-join operation using the JOIN operator.
grunt> Relation3_name = JOIN Relation1_name BY key, Relation2_name BY key ;
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Here is the syntax of performing inner join operation using the JOIN operator.
grunt> result = JOIN relation1 BY columnname, relation2 BY columnname;
Pig Latin Script for Filter:
The FILTER operator is used to select the required tuples from a relation based on a
condition.
Syntax:
grunt> Relation2_name = FILTER Relation1_name BY (condition);
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Program-9
Aim: Run the Pig Latin Scripts to find Word Count.
Program:
Assume we have data in the file like below:
This is a hadoop post
hadoop is a bigdata technology
we want to generate output for count of each word like below:
(a,2)
(is,2)
(This,1)
(class,1)
(hadoop,2)
(bigdata,1)
(technology,1)
Steps how to generate the same using Pig Latin:
Step 1: Load the data from HDFS
Use Load statement to load the data into a relation.
As keyword used to declare column names, as we don’t have any columns, we declared only
one column named line.
input = LOAD '/path/to/file/' AS(line:Chararray);
Step 2: Convert the Sentence into words
The data we have is in sentences. So we have to convert that data into words using
TOKENIZE Function.
(TOKENIZE(line));
(or)
If we have any delimeter like space we can specify as
(TOKENIZE(line,' '));
Output will be like this:
({(This),(is),(a),(hadoop),(class)})
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
({(hadoop),(is),(a),(bigdata),(technology)})
but we have to convert it into multiple rows like below:
(This)
(is)
(a)
(hadoop)
(class)
(hadoop)
(is)
(a)
(bigdata)
(technology)
Convert Column into Rows
I mean we have to convert every line of data into multiple rows ,for this we have function
called
FLATTEN in pig.
Using FLATTEN function the bag is converted into tuple, means the array of strings
converted into multiple rows.
Words = FOREACH input GENERATE FLATTEN(TOKENIZE(line,' ')) AS word;
Then the ouput is like below
(This)
(is)
(a)
(hadoop)
(class)
(hadoop)
(is)
(a)
(bigdata)
(technology)
Step 3: Apply GROUP BY
We have to count each word occurance, for that we have to group all the words.
Grouped = GROUP words BY word;
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Step 4: Generate word count
wordcount = FOREACH Grouped GENERATE group, COUNT(words);
We can print the word count on console using Dump.
DUMP wordcount;
Output will be like below:
(a,2)
(is,2)
(This,1)
(class,1)
(hadoop,2)
(bigdata,1)
(technology,1)
Below is the complete program for the same:
input = LOAD '/path/to/file/' AS(line:Chararray);
Words = FOREACH input GENERATE FLATTEN(TOKENIZE(line,' ')) AS word;
Grouped = GROUP words BY word;
wordcount = FOREACH Grouped GENERATE group, COUNT(words);
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
Program-10
Aim: Run the Pig Latin Scripts to find a max temp for each and every year.
Word Count using Pig Latin Steps:
Step1:
1. Create a text file having few lines of text and save it as bd.txt.
2. Create on directory in hdfs named wc
3. Store bd..txt file from local file system to hdfs file system directory wc
Step2:
inputline = load '/user/cloudera/wc/bd.txt' using PigStorage('\t') as (data:chararray);
words = FOREACH inputline GENERATE FLATTEN(TOKENIZE(data)) AS word;
filtered_words = FILTER words BY word MATCHES '\\w+';
word_groups = GROUP filtered_words BY word;
word_count = FOREACH word_groups GENERATE COUNT(filtered_words) AS
count, group AS word;
ordered_word_count = ORDER word_count BY count DESC;
DUMP ordered_word_count;
You can use the below command to save the result in HDFS:
grunt> store ordered_word_count; into '/user/cloudera/wc/output/';
Find the Maximum Year in a given dataset using PIG
Option1:
A = LOAD 'input' USING PigStorage() AS (Year:int,Temp:int);
B = GROUP A ALL;
C = FOREACH B GENERATE MAX(A.Temp);
DUMP C;
[Approved by AICTE, Govt. of India & Affiliated to Dr. APJ
GL BAJAJ
Institute of Technologies & Management
Abdul Kalam Technical University, Lucknow, U.P., India]
Department of Applied Computational Science & Engineering
Greater Noida
or
Option2: Using (ORDER and LIMIT)
A = LOAD 'input' USING PigStorage() AS (Year:int,Temp:int);
B = ORDER A BY Temp DESC;
C = LIMIT B 1;
D = FOREACH C GENERATE Temp;
DUMP D;
