Hadoop 2.2.0 버전의 기준으로 작성 되었습니다.
1. hadoop 사용자 추가
[root@ruo91 ~]# useradd hadoop [root@ruo91 ~]# passwd hadoop
2. SSH 설정
로컬 SSH 접속시 비밀번호 요구 하지 않도록 설정
[root@ruo91 ~]# su - hadoop [hadoop@ruo91 ~]$ ssh-keygen -t dsa -P '' -f ~/.ssh/id_dsa [hadoop@ruo91 ~]$ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys [hadoop@ruo91 ~]$ chmod 644 ~/.ssh/authorized_keys
– 접속 테스트
[hadoop@ruo91 ~]$ ssh localhost Last login: Thu Oct 17 11:49:09 2013 from localhost.localdomain Hello?
3. JDK 설치
http://www.oracle.com/technetwork/java/javase/downloads/index.html
[hadoop@ruo91 ~]$ curl -LO "http://download.oracle.com/otn-pub/java/jdk/7u55-b13/jdk-7u55-linux-x64.tar.gz" \ -H 'Cookie: oraclelicense=accept-securebackup-cookie' [hadoop@ruo91 ~]$ tar xzvf jdk-7u55-linux-x64.tar.gz [hadoop@ruo91 ~]$ mv jdk1.7.0_55 jdk
– 환경 변수 등록
[hadoop@ruo91 ~]$ nano ~/.bash_profile # JAVA export JAVA_HOME=$HOME/jdk export PATH=$PATH:$JAVA_HOME/bin
4. Hadoop 설치
http://www.apache.org/dyn/closer.cgi/hadoop/common/
[hadoop@ruo91 ~]$ wget http://mirror.apache-kr.org/hadoop/core/stable2/hadoop-2.2.0.tar.gz [hadoop@ruo91 ~]$ tar xzvf hadoop-2.2.0.tar.gz [hadoop@ruo91 ~]$ mv hadoop-2.2.0 2.2.0
– Hadoop 환경 변수 등록
[hadoop@ruo91 ~]$ nano ~/.bash_profile
# Hadoop
export HADOOP_PREFIX="$HOME/2.2.0"
export PATH=$PATH:$HADOOP_PREFIX/bin
export PATH=$PATH:$HADOOP_PREFIX/sbin
export HADOOP_MAPRED_HOME=${HADOOP_PREFIX}
export HADOOP_COMMON_HOME=${HADOOP_PREFIX}
export HADOOP_HDFS_HOME=${HADOOP_PREFIX}
export YARN_HOME=${HADOOP_PREFIX}
– hadoop-env.sh
[hadoop@ruo91 ~]$ nano 2.2.0/etc/hadoop/hadoop-env.sh
export JAVA_HOME=$HOME/jdk
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"
– yarn-env.sh
[hadoop@ruo91 ~]$ nano 2.2.0/etc/hadoop/yarn-env.sh
export HADOOP_COMMON_LIB_NATIVE_DIR=${HADOOP_PREFIX}/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_PREFIX/lib"
– 환경변수 적용
[hadoop@ruo91 ~]$ source ~/.bash_profile
5. Hadoop 설정
– core-site.xml
[hadoop@ruo91 ~]$ nano $HADOOP_PREFIX/etc/hadoop/core-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
<final>true</final>
</property>
</configuration>
– hdfs-site.xml
[hadoop@ruo91 ~]$ nano $HADOOP_PREFIX/etc/hadoop/hdfs-site.xml
<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/home/hadoop/dfs/name</value>
<final>true</final>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/home/hadoop/dfs/data</value>
<final>true</final>
</property>
<property>
<name>dfs.permissions</name>
<value>false</value>
</property>
</configuration>
– mapred-site.xml
[hadoop@ruo91 ~]$ nano $HADOOP_PREFIX/etc/hadoop/mapred-site.xml
<?xml version="1.0"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<!-- Put site-specific property overrides in this file. -->
<configuration>
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
<property>
<name>mapred.system.dir</name>
<value>file:/home/hadoop/mapred/system</value>
<final>true</final>
</property>
<property>
<name>mapred.local.dir</name>
<value>file:/home/hadoop/mapred/local</value>
<final>true</final>
</property>
</configuration>
– yarn-site.xml
[hadoop@ruo91 ~]$ nano $HADOOP_PREFIX/etc/hadoop/yarn-site.xml
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<configuration>
<!-- Site specific YARN configuration properties -->
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
</property>
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
</configuration>
6. Hadoop 실행
– Name Node format
[hadoop@ruo91 ~]$ hdfs namenode -format 13/10/17 23:18:29 INFO namenode.NameNode: STARTUP_MSG: /************************************************************ STARTUP_MSG: Starting NameNode STARTUP_MSG: host = ruo91.yongbok.net/127.0.0.1 STARTUP_MSG: args = [-format] STARTUP_MSG: version = 2.2.0 STARTUP_MSG: classpath = /home/hadoop/2.2.0/etc/hadoop:/home/hadoop/2.2.0/share/hadoop/common/lib/jettison- ........ .................... .......................... Blah blah ~ ................................ 2.2.0.jar:/home/hadoop/2.2.0/contrib/capacity-scheduler/*.jar STARTUP_MSG: build = https://svn.apache.org/repos/asf/hadoop/common -r 1529768; compiled by 'hortonmu' on 2013-10-07T06:28Z STARTUP_MSG: java = 1.7.0_45 ************************************************************/ 13/10/17 23:18:29 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT] Java HotSpot(TM) 64-Bit Server VM warning: You have loaded library /home/hadoop/2.2.0/lib/native/libhadoop.so.1.0.0 which might have disabled stack guard. The VM will try to fix the stack guard now. It's highly recommended that you fix the library with 'execstack -c ', or link it with '-z noexecstack'. 13/10/17 23:18:30 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Formatting using clusterid: CID-060a7239-4bac-4801-a720-9702eb60c341 13/10/17 23:18:31 INFO namenode.HostFileManager: read includes: HostSet( ) 13/10/17 23:18:31 INFO namenode.HostFileManager: read excludes: HostSet( ) 13/10/17 23:18:31 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000 13/10/17 23:18:31 INFO util.GSet: Computing capacity for map BlocksMap 13/10/17 23:18:31 INFO util.GSet: VM type = 64-bit 13/10/17 23:18:31 INFO util.GSet: 2.0% max memory = 966.7 MB 13/10/17 23:18:31 INFO util.GSet: capacity = 2^21 = 2097152 entries 13/10/17 23:18:31 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false 13/10/17 23:18:31 INFO blockmanagement.BlockManager: defaultReplication = 1 13/10/17 23:18:31 INFO blockmanagement.BlockManager: maxReplication = 512 13/10/17 23:18:31 INFO blockmanagement.BlockManager: minReplication = 1 13/10/17 23:18:31 INFO blockmanagement.BlockManager: maxReplicationStreams = 2 13/10/17 23:18:31 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks = false 13/10/17 23:18:31 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000 13/10/17 23:18:31 INFO blockmanagement.BlockManager: encryptDataTransfer = false 13/10/17 23:18:31 INFO namenode.FSNamesystem: fsOwner = hadoop (auth:SIMPLE) 13/10/17 23:18:31 INFO namenode.FSNamesystem: supergroup = supergroup 13/10/17 23:18:31 INFO namenode.FSNamesystem: isPermissionEnabled = false 13/10/17 23:18:31 INFO namenode.FSNamesystem: HA Enabled: false 13/10/17 23:18:31 INFO namenode.FSNamesystem: Append Enabled: true 13/10/17 23:18:32 INFO util.GSet: Computing capacity for map INodeMap 13/10/17 23:18:32 INFO util.GSet: VM type = 64-bit 13/10/17 23:18:32 INFO util.GSet: 1.0% max memory = 966.7 MB 13/10/17 23:18:32 INFO util.GSet: capacity = 2^20 = 1048576 entries 13/10/17 23:18:32 INFO namenode.NameNode: Caching file names occuring more than 10 times 13/10/17 23:18:32 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033 13/10/17 23:18:32 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0 13/10/17 23:18:32 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension = 30000 13/10/17 23:18:32 INFO namenode.FSNamesystem: Retry cache on namenode is enabled 13/10/17 23:18:32 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis 13/10/17 23:18:32 INFO util.GSet: Computing capacity for map Namenode Retry Cache 13/10/17 23:18:32 INFO util.GSet: VM type = 64-bit 13/10/17 23:18:32 INFO util.GSet: 0.029999999329447746% max memory = 966.7 MB 13/10/17 23:18:32 INFO util.GSet: capacity = 2^15 = 32768 entries 13/10/17 23:18:32 INFO common.Storage: Storage directory /home/hadoop/dfs/name has been successfully formatted. 13/10/17 23:18:32 INFO namenode.FSImage: Saving image file /home/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression 13/10/17 23:18:32 INFO namenode.FSImage: Image file /home/hadoop/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 198 bytes saved in 0 seconds. 13/10/17 23:18:32 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0 13/10/17 23:18:32 INFO util.ExitUtil: Exiting with status 0 13/10/17 23:18:32 INFO namenode.NameNode: SHUTDOWN_MSG: /************************************************************ SHUTDOWN_MSG: Shutting down NameNode at ruo91.yongbok.net/127.0.0.1 ************************************************************/
– Hadoop (name node, data node, yarn) 실행
[hadoop@ruo91 ~]$ start-all.sh
– 프로세스 확인
[hadoop@ruo91 ~]$ jps 32267 SecondaryNameNode 32028 DataNode 415 NodeManager 31905 NameNode 452 Jps 319 ResourceManager
– 웹 확인
http://localhost:8088/ or http://localhost:50070/
hadoop_cluster
hadoop_admin
