Downloaded 237 times
![Format a new system
notroot@ubuntu:/usr/local/hadoop/conf$ su - hduser
Password:
hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format
13/04/03 13:41:24 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ubuntu.localdomain/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290;
compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
Re-format filesystem in /hadoop_data/name ? (Y or N) Y
13/04/03 13:41:26 INFO util.GSet: VM type = 32-bit
13/04/03 13:41:26 INFO util.GSet: 2% max memory = 19.33375 MB
13/04/03 13:41:26 INFO util.GSet: capacity = 2^22 = 4194304 entries
….
13/04/03 13:41:28 INFO common.Storage: Storage directory /hadoop_data/name has been successfully formatted.
13/04/03 13:41:28 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu.localdomain/127.0.1.1
************************************************************/
Do not format a running Hadoop file system as you will lose all the
data currently in the cluster (in HDFS)!](https://image.slidesharecdn.com/hadoopsingleclusterinstallation-130527124305-phpapp01/85/Hadoop-single-cluster-installation-18-320.jpg)
![Format a new system
notroot@ubuntu:/usr/local/hadoop/conf$ su - hduser
Password:
hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format
13/04/03 13:41:24 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = ubuntu.localdomain/127.0.1.1
STARTUP_MSG: args = [-format]
STARTUP_MSG: version = 1.0.4
STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290;
compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012
************************************************************/
Re-format filesystem in /hadoop_data/name ? (Y or N) Y
13/04/03 13:41:26 INFO util.GSet: VM type = 32-bit
13/04/03 13:41:26 INFO util.GSet: 2% max memory = 19.33375 MB
13/04/03 13:41:26 INFO util.GSet: capacity = 2^22 = 4194304 entries
….
13/04/03 13:41:28 INFO common.Storage: Storage directory /hadoop_data/name has been successfully formatted.
13/04/03 13:41:28 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at ubuntu.localdomain/127.0.1.1
************************************************************/
Do not format a running Hadoop file system as you will lose all the
data currently in the cluster (in HDFS)!](https://image.slidesharecdn.com/hadoopsingleclusterinstallation-130527124305-phpapp01/75/Hadoop-single-cluster-installation-18-2048.jpg)
Uploaded byMinh Tran
Hadoop single cluster installation
The document details the steps for installing Hadoop on an Ubuntu server, including prerequisites such as installing JDK, SSH, and Hadoop itself. It describes how to configure the environment, start a single-node cluster, run a MapReduce job using example eBooks, and check the Hadoop process status. Configuration files and command examples are provided for clarity throughout the installation process.
More Related Content
Similar to Hadoop single cluster installation
![UiPath Automation Suite Installation (Hands-On) [2/3]](https://cdn.slidesharecdn.com/ss_thumbnails/automationsuitecommunitysession2-251015095633-a6d862f1-thumbnail.jpg?width=600ounds&width=560&fit=bounds)
Hadoop single cluster installation
- 1. Hadoop Single Cluster Installation MinhTran – Software Architect 05/2013
- 2. Prerequisites • Ubuntu Server10.04 (Lucid Lynx) • JDK 6u34 Linux • Hadoop 1.0.4 • VMWare Player / VMWare Workstation / VMWare Server • Ubuntu Server VMWare Image: http://www.thoughtpolice.co.uk/vmware/#u buntu10.04 (notroot / thoughtpolice)
- 3. Install SSH • sudoapt-get update • sudo apt-get install openssh-server
- 4. Install JDK • wget-c -O jdk-6u34-linux-i586.bin http://download.oracle.com/otn/java/jdk/6u34-b04/jdk-6u34- linux- i586.bin?AuthParam=1347897296_c6dd13e0af9e099dc731937f9 5c1cd01 • chmod 777 jdk-6u34-linux-i586.bin • ./jdk-6u34-linux-i586.bin • sudo mv jdk1.6.0_34 /usr/local • sudo ln -s /usr/local/jdk1.6.0_34 /usr/local/jdk
- 5. Create group /account for Hadoop • sudo addgroup hadoop • sudo adduser --ingroup hadoop hduser
- 6. Install Local Hadoop •wget http://mirrors.digipower.vn/apache/hadoop/common/hado op-1.0.4/hadoop-1.0.4.tar.gz • tar -zxvf hadoop-1.0.4.tar.gz • sudo mv hadoop-1.0.4 /usr/local • sudo chown -R hduser:hadoop /usr/local/hadoop-1.0.4 • sudo ln -s /usr/local/hadoop-1.0.4 /usr/local/hadoop
- 7. Install Apache Ant •wget http://mirrors.digipower.vn/apache/ant/binaries/apache- ant-1.9.0-bin.tar.gz • tar -zxvf apache-ant-1.9.0-bin.tar.gz • sudo mv apache-ant-1.9.0 /usr/local • sudo ln -s /usr/local/apache-ant-1.9.0 /usr/local/apache-ant
- 8. Modify environment variables •su - hduser • vi .bashrc • export JAVA_HOME=/usr/local/jdk • export HADOOP_PREFIX=/usr/local/hadoop • export PATH=${JAVA_HOME}/bin:${HADOOP_PREFIX}/bin:${PATH} • . .bashrc
- 9. Try 1st example hduser@ubuntu:/usr/local/hadoop$cd $HADOOP_PREFIX hduser@ubuntu:/usr/local/hadoop$ hadoop jar hadoop-examples-1.0.4.jar pi 2 10 Number of Maps = 2 Samples per Map = 10 Wrote input for Map #0 Wrote input for Map #1 Starting Job 13/04/03 15:01:40 INFO mapred.FileInputFormat: Total input paths to process : 2 13/04/03 15:01:41 INFO mapred.JobClient: Running job: job_201304031458_0003 13/04/03 15:01:42 INFO mapred.JobClient: map 0% reduce 0% 13/04/03 15:02:00 INFO mapred.JobClient: map 100% reduce 0% 13/04/03 15:02:15 INFO mapred.JobClient: map 100% reduce 100% 13/04/03 15:02:19 INFO mapred.JobClient: Job complete: job_201304031458_0003 13/04/03 15:02:19 INFO mapred.JobClient: Counters: 30 13/04/03 15:02:19 INFO mapred.JobClient: Job Counters … 13/04/03 15:02:19 INFO mapred.JobClient: Reduce output records=0 13/04/03 15:02:19 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1118670848 13/04/03 15:02:19 INFO mapred.JobClient: Map output records=4 Job Finished in 39.148 seconds Estimated value of Pi is 3.80000000000000000000
- 10. Setup Single NodeCluster • Disabling ipv6 • Configuring SSH • Configuration – hadoop-env.sh – conf/*-site.xml • Start / Stop node cluster • Running MapReduce job
- 11. Disabling ipv6 • Open/etc/sysctl.conf, add following lines # disable ipv6 net.ipv6.conf.all.disable_ipv6 = 1 net.ipv6.conf.default.disable_ipv6 = 1 net.ipv6.conf.lo.disable_ipv6 = 1 • Reboot your machine • Verify ipv6 enabled / disabled cat /proc/sys/net/ipv6/conf/all/disable_ipv6 (0 – enabled, 1 – disabled)
- 12. Configuring SSH • CreateSSH keys in the localhost su - hduser ssh-keygen -t rsa -P "“ • Put the key id_rsa.pub to localhost touch ~/.ssh/authorized_keys && chmod 600 ~/.ssh/authorized_keys cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
- 13. Configuration • Edit theconfiguration in /usr/local/hadoop/conf/hadoop- env.sh, add following lines: export JAVA_HOME=/usr/local/jdk
- 14. Configuration (cont.) • Createa folder to store data for node sudo mkdir -p /hadoop_data/name sudo mkdir -p /hadoop_data/data sudo mkdir -p /hadoop_data/temp sudo chown hduser:hadoop /hadoop_data/name sudo chown hduser:hadoop /hadoop_data/data sudo chown hduser:hadoop /hadoop_data/temp
- 15. conf/core-site.xml <configuration> <property> <name>hadoop.tmp.dir</name> <value>/hadoop_data/temp</value> <description>A base forother temporary directories.</description> </property> <property> <name>fs.default.name</name> <value>hdfs://localhost:54310</value> <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description> </property> </configuration>
- 16. conf/mapred-site.xml <configuration> <property> <name>mapred.job.tracker</name> <value>localhost:54311</value> <description>The host andport that the MapReduce job tracker runs at. If "local", then jobs are run in-process as a single map and reduce task. </description> </property> </configuration>
- 17. conf/hdfs-site.xml <configuration> <property> <name>dfs.name.dir</name> <!-- Path tostore namespace and transaction logs --> <value>/hadoop_data/name</value> </property> <property> <name>dfs.data.dir</name> <!-- Path to store data blocks in datanode --> <value>/hadoop_data/data</value> </property> <property> <name>dfs.replication</name> <value>1</value> <description>Default block replication. The actual number of replications can be specified when the file is created. The default is used if replication is not specified in create time. </description> </property> </configuration>
- 18. Format a newsystem notroot@ubuntu:/usr/local/hadoop/conf$ su - hduser Password: hduser@ubuntu:~$ /usr/local/hadoop/bin/hadoop namenode -format 13/04/03 13:41:24 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = ubuntu.localdomain/127.0.1.1 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 1.0.4 STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common/branches/branch-1.0 -r 1393290; compiled by 'hortonfo' on Wed Oct 3 05:13:58 UTC 2012 ************************************************************/ Re-format filesystem in /hadoop_data/name ? (Y or N) Y 13/04/03 13:41:26 INFO util.GSet: VM type = 32-bit 13/04/03 13:41:26 INFO util.GSet: 2% max memory = 19.33375 MB 13/04/03 13:41:26 INFO util.GSet: capacity = 2^22 = 4194304 entries …. 13/04/03 13:41:28 INFO common.Storage: Storage directory /hadoop_data/name has been successfully formatted. 13/04/03 13:41:28 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at ubuntu.localdomain/127.0.1.1 ************************************************************/ Do not format a running Hadoop file system as you will lose all the data currently in the cluster (in HDFS)!
- 19. Start Single NodeCluster hduser@ubuntu:~$ /usr/local/hadoop/bin/start-all.sh starting namenode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser- namenode-ubuntu.out localhost: starting datanode, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop- hduser-datanode-ubuntu.out localhost: starting secondarynamenode, logging to /usr/local/hadoop- 1.0.4/libexec/../logs/hadoop-hduser-secondarynamenode-ubuntu.out starting jobtracker, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop-hduser- jobtracker-ubuntu.out localhost: starting tasktracker, logging to /usr/local/hadoop-1.0.4/libexec/../logs/hadoop- hduser-tasktracker-ubuntu.out
- 20. How to verifyHadoop process • A nifty tool for checking whether the expected Hadoop processes are running is jps (part of Sun JDK tool) hduser@ubuntu:~$ jps 1203 NameNode 1833 Jps 1615 JobTracker 1541 SecondaryNameNode 1362 DataNode 1788 TaskTracker • You can also check with netstat if Hadoop is listening on the configured ports. notroot@ubuntu:/usr/local/hadoop/conf$ sudo netstat -plten | grep java tcp 0 0 127.0.0.1:54310 0.0.0.0:* LISTEN 1001 7167 2438/java tcp 0 0 127.0.0.1:54311 0.0.0.0:* LISTEN 1001 7949 2874/java tcp 0 0 0.0.0.0:50090 0.0.0.0:* LISTEN 1001 7898 2791/java tcp 0 0 0.0.0.0:50030 0.0.0.0:* LISTEN 1001 8035 2874/java tcp 0 0 0.0.0.0:50070 0.0.0.0:* LISTEN 1001 7202 2438/java tcp 0 0 0.0.0.0:57143 0.0.0.0:* LISTEN 1001 7585 2791/java tcp 0 0 0.0.0.0:41943 0.0.0.0:* LISTEN 1001 7222 2608/java tcp 0 0 0.0.0.0:58936 0.0.0.0:* LISTEN 1001 6969 2438/java tcp 0 0 127.0.0.1:50234 0.0.0.0:* LISTEN 1001 8158 3050/java tcp 0 0 0.0.0.0:50010 0.0.0.0:* LISTEN 1001 7697 2608/java tcp 0 0 0.0.0.0:50075 0.0.0.0:* LISTEN 1001 7775 2608/java tcp 0 0 0.0.0.0:40067 0.0.0.0:* LISTEN 1001 7764 2874/java tcp 0 0 0.0.0.0:50020 0.0.0.0:* LISTEN 1001 7939 2608/java
- 21. Stop your singlenode cluster hduser@ubuntu:~$ /usr/local/hadoop/bin/stop-all.sh stopping jobtracker localhost: stopping tasktracker stopping namenode localhost: stopping datanode localhost: stopping secondarynamenode
- 22. Running a MapReducejob • We will use three ebooks from Project Gutenberg for this example: – The Outline of Science, Vol. 1 (of 4) by J. Arthur Thomson – The Notebooks of Leonardo Da Vinci – Ulysses by James Joyce • Download each ebook as text files in Plain Text UTF-8 encoding and store the files in /tmp/gutenberg
- 23. Running a MapReducejob (cont.) • Copy these files into HDFS hduser@ubuntu:~$ hadoop dfs -copyFromLocal /tmp/gutenberg /user/hduser/gutenberg hduser@ubuntu:~$ hadoop dfs -ls /user/hduser/gutenberg Found 3 items -rw-r--r-- 1 hduser supergroup 661807 2013-04-03 14:01 /user/hduser/gutenberg/pg20417.txt -rw-r--r-- 1 hduser supergroup 1540092 2013-04-03 14:01 /user/hduser/gutenberg/pg4300.txt -rw-r--r-- 1 hduser supergroup 1391684 2013-04-03 14:01 /user/hduser/gutenberg/pg5000.txt
- 24. Running a MapReducejob (cont.) hduser@ubuntu:~$ cd /usr/local/hadoop hduser@ubuntu:/usr/local/hadoop$ hadoop jar hadoop-examples-1.0.4.jar wordcount /user/hduser/gutenberg /user/hduser/gutenberg-output 13/04/03 14:02:45 INFO input.FileInputFormat: Total input paths to process : 3 13/04/03 14:02:45 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/04/03 14:02:45 WARN snappy.LoadSnappy: Snappy native library not loaded 13/04/03 14:02:45 INFO mapred.JobClient: Running job: job_201304031352_0001 13/04/03 14:02:46 INFO mapred.JobClient: map 0% reduce 0% 13/04/03 14:03:09 INFO mapred.JobClient: map 66% reduce 0% 13/04/03 14:03:32 INFO mapred.JobClient: map 100% reduce 0% 13/04/03 14:03:47 INFO mapred.JobClient: map 100% reduce 100% 13/04/03 14:03:53 INFO mapred.JobClient: Job complete: job_201304031352_0001 13/04/03 14:03:53 INFO mapred.JobClient: Counters: 29 13/04/03 14:03:53 INFO mapred.JobClient: Job Counters 13/04/03 14:03:53 INFO mapred.JobClient: Launched reduce tasks=1 … 13/04/03 14:03:53 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=59114 13/04/03 14:03:53 INFO mapred.JobClient: SPLIT_RAW_BYTES=361 13/04/03 14:03:53 INFO mapred.JobClient: Reduce input records=102321 13/04/03 14:03:53 INFO mapred.JobClient: Reduce input groups=82334 13/04/03 14:03:53 INFO mapred.JobClient: Combine output records=102321 13/04/03 14:03:53 INFO mapred.JobClient: Physical memory (bytes) snapshot=576069632 13/04/03 14:03:53 INFO mapred.JobClient: Reduce output records=82334 13/04/03 14:03:53 INFO mapred.JobClient: Virtual memory (bytes) snapshot=1490481152 13/04/03 14:03:53 INFO mapred.JobClient: Map output records=629172
- 25. Check the result •hduser@ubuntu:/usr/local/hadoop$ hadoop dfs -ls /user/hduser/gutenberg-output Found 3 items -rw-r--r-- 1 hduser supergroup 0 2013-04-03 14:03 /user/hduser/gutenberg-output/_SUCCESS drwxr-xr-x - hduser supergroup 0 2013-04-03 14:02 /user/hduser/gutenberg-output/_logs -rw-r--r-- 1 hduser supergroup 880829 2013-04-03 14:03 /user/hduser/gutenberg-output/part-r-00000 • hduser@ubuntu:/usr/local/hadoop$ hadoop dfs -cat /user/hduser/gutenberg-output/part-r-00000 | more "(Lo)cra" 1 "1490 1 "1498," 1 "35" 1 "40," 1 "A 2 "AS-IS". 1 "A_ 1 "Absoluti 1 "Alack! 1 "Alack!" 1 "Alla 1 "Allegorical 1 "Alpha 1 "Alpha," 1 …
- 26. Hadoop Interfaces • NameNodeWeb UI: http://192.168.65.134:50070/ • JobTracker Web UI: http://192.168.65.134:50030/ • TaskTracker Web UI: http://192.168.65.134:50060/
- 27. NameNode Web UIdaemon
- 28.
- 29.
- 30. Troubleshooting • VMware Ubuntuimage lost eth0 after moving it http://www.whiteboardcoder.com/2012/03/vmware-ubuntu-image- lost-eth0-after.html • Hadoop Troubleshooting: http://wiki.apache.org/hadoop/TroubleShooting • Error when formatting the Hadoop filesystem: http://askubuntu.com/questions/35551/error-when- formatting-the-hadoop-filesystem
- 31.
Editor's Notes
- #11 In the preceding sample, MapReduce worked in the local mode without starting any servers and using the local filesystem as the storage system for inputs, outputs, and working data. The following diagram shows what happened in the WordCount program under the covers: