SRM INSTITUTE OF SCIENCE AND TECHNOLOGY: VADAPALANI CAMPUS
DEPARTMENT OF COMPUTER SCIENCE AND ENGINEEERING
Prerequisite
ubuntu 16.04
Make ec2 as password Authentication:
use command for setting password to ec2 ubuntu image: sudo passwd ubuntu
Step:1 JAVA 8-----
1. sudo add-apt-repository ppa:webupd8team/java
2. sudo apt-get update
3. sudo apt-get install oracle-java8-installer
4. sudo apt-get install oracle-java8-set-default
Step 2: SSH SERVER INSTALLATION
5. sudo apt-get install openssh-server
6. sudo sed -i -e 's/PasswordAuthentication no/PasswordAuthentication yes/g'
/etc/ssh/sshd_config
7. ssh-keygen -t dsa -P “” -f ~/.ssh/id_dsa
cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys
8. sudo service ssh restart
9. ssh localhost
//passwordless login
10. exit
Step 3: Download hadoop package
https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-2.7.3.tar.gz
10 .wget https://archive.apache.org/dist/hadoop/core/hadoop-2.7.3/hadoop-
2.7.3.tar.gz
11. sudo tar -xzvf hadoop-2.7.3.tar.gz
sudo mkdir -p /usr/local/hadoop
sudo mv hadoop-2.7.3/* /usr/local/hadoop/
12. sudo chown -R ubuntu:ubuntu /usr/local/hadoop
//create folder for datanode and name node
13sudo mkdir -p /app/hadoop/tmp
14 sudo mkdir -p /app/hadoop/tmp
set permission
15 sudo chown -R ubuntu /app/hadoop/tmp
Step 4: Configure Hadoop:
Check where your Java is installed:
16 readlink -f /usr/bin/java
If you get something like /usr/lib/jvm/java-8-oracle/jre/bin/java,
/usr/lib/jvm/java-8-oracle is what you should used for JAVA_HOME.
Add to ~/.bashrc file:
17 sudo nano ~/.bashrc
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib/native"
Reload ~/.bashrc file:
18 source ~/.bashrc
Modify JAVA_HOME in
19 sudo nano /usr/local/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-8-oracle
Modify
20. sudo nano /usr/local/hadoop/etc/hadoop/core-site.xml
to have something like:
<configuration>
...
<property>
<name>fs.default.name</name>
<value>hdfs://master:9000</value>
</property>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
...
</configuration>
Modify
21. sudo nano /usr/local/hadoop/etc/hadoop/yarn-site.xml
to have something like:
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>master:8040</value>
</property>
Create /usr/local/lib/hadoop-2.7.0/etc/hadoop/mapred-site.xml
from template:
21. cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template
/usr/local/hadoop/etc/hadoop/mapred-site.xml
Modify
22. sudo nano /usr/local/hadoop/etc/hadoop/mapred-site.xml
to have something like:
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
</configuration>
Modify
23. sudo nano /usr/local/hadoop/etc/hadoop/hdfs-site.xml
to have something like:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>
Format file system:
24. hdfs namenode -format
Start Hadoop:
25. start-dfs.sh
26. start-yarn.sh
You might be asked to accept machine’s key.
Check if everything is running:
27. jps
You should get something like:
Jps
NodeManager
NameNode
ResourceManager
DataNode
SecondaryNameNode
TYPE IN WEB BROWSER
28. http://localhost:8088/cluster
29. http://localhost:50070/
INSTALLED HADOOP CLUSTER SUCCESSFULLY IN AMAZON EC2
