いまさらながらHadoopを使ってみた

CentOS 6.5にHadoopをインストールして使ってみた記録

(1)CentOSのインストール
インストール構成を選ぶ画面で「Software Development Workstation」を選んだ以外はほぼでフォルトでインストール。
これで初回起動時にJavaがインストール済み。

JAVA_HOMEのパスを通しておく。/etc/profileのいちばん下に追記する。
export JAVA_HOME=/usr/lib/jvm/java-1.7.0-openjdk.x86_64
export PATH=$PATH:$JAVA_HOME/bin
export CLASSPATH=.:$JAVA_HOME/jre/lib:$JAVA_HOME/lib:$JAVA_HOME/lib/tools.jar

↑実を言うとこれではサービスを起動したときにJAVAのパスが通ってない旨のメッセージが出た。
+======================================================================+
| Error: JAVA_HOME is not set and Java could not be found |
+----------------------------------------------------------------------+
| Please download the latest Sun JDK from the Sun Java web site |
| > http://java.sun.com/javase/downloads/ < |
| |
| Hadoop requires Java 1.6 or later. |
| NOTE: This script will find Sun Java whether you install using the |
| binary or the RPM based installer. |
+======================================================================+
+======================================================================+
| Error: JAVA_HOME is not set and Java could not be found |
+----------------------------------------------------------------------+
| Please download the latest Sun JDK from the Sun Java web site |
| > http://java.sun.com/javase/downloads/ < |
| |
| Hadoop requires Java 1.6 or later. |
| NOTE: This script will find Sun Java whether you install using the |
| binary or the RPM based installer. |
+======================================================================+
+======================================================================+
| Error: JAVA_HOME is not set and Java could not be found |
+----------------------------------------------------------------------+
| Please download the latest Sun JDK from the Sun Java web site |
| > http://java.sun.com/javase/downloads/ < |
| |
| Hadoop requires Java 1.6 or later. |
| NOTE: This script will find Sun Java whether you install using the |
| binary or the RPM based installer. |
+======================================================================+
+======================================================================+
| Error: JAVA_HOME is not set and Java could not be found |
+----------------------------------------------------------------------+
| Please download the latest Sun JDK from the Sun Java web site |
| > http://java.sun.com/javase/downloads/ < |
| |
| Hadoop requires Java 1.6 or later. |
| NOTE: This script will find Sun Java whether you install using the |
| binary or the RPM based installer. |
+======================================================================+

そんなわけで,jdk-8-linux-i586.rpm を取ってきて
rpm -ivh jdk-8-linux-i586.rpm

(2)Hadoopのインストール
cd /etc/yum.repos.d
wget http://archive.cloudera.com/redhat/cdh/cloudera-cdh3.repo
yum install hadoop-0.20

このあたりもインストールしておく。
hadoop-0.20-jobtracker.noarch 0:0.20.2+923.97-1
hadoop-0.20-secondarynamenode.noarch 0:0.20.2+923.97-1
hadoop-0.20-namenode.noarch 0:0.20.2+923.97-1
hadoop-0.20-datanode.noarch 0:0.20.2+923.97-1
hadoop-0.20-tasktracker.noarch 0:0.20.2+923.97-1

スレーブの場合、datanodeとtasktrackerだけでいいのかも。


(3)Hadoopの設定ファイル一式を作成
cp -r /etc/hadoop-0.20/conf.empty /etc/hadoop-0.20/conf.test1
alternatives --install /etc/hadoop-0.20/conf hadoop-0.20-conf /etc/hadoop-0.20/conf.test1 20
alternatives --display hadoop-0.20-conf

(4)service-start.shとか作っておいて起動する
service hadoop-0.20-namenode restart
service hadoop-0.20-jobtracker restart
service hadoop-0.20-datanode restart
service hadoop-0.20-tasktracker restart

(5)SELinuxおよびiptableの無効化

$vi /etc/selinux/config と
$vi /etc/sysconfig/selinux で
SELinuxを有効にする場合はSELINUX=enforcing
SELinuxを無効にする場合はSELINUX=disabled

$ /etc/rc.d/init.d/iptables stop
$ chkconfig iptables off
$ chkconfig --list iptables
iptables 0:off 1:off 2:off 3:off 4:off 5:off 6:off

1台構成のHadoopを30分で試してみる(CentOS + Cloudera)
http://saburi380.blogspot.jp/2009/11/301hadoopcentos-cloudera.html

4台構成のHadoopを100分で試してみる(CentOS + Cloudera)
http://saburi380.blogspot.jp/2009/11/1004hadoopcentos-cloudera.html

mecab関連
http://mitsuakikawamorita.com/blog/?p=671
http://lxyuma.hatenablog.com/entry/20121128/1354107593

タグ:

コメント(0)

コメントを投稿する