mac에서 하둡 hdfs 설치 및 실행하기

1. homebrew를 통해 hadoop package 설치

$ brew install hadoop

hadoop이 설치된 경로는 아래와 같이 명령하여 확인할 수 있다.

$ brew info hadoop
hadoop: stable 3.3.0
Framework for distributed processing of large data sets
https://hadoop.apache.org/
Conflicts with:
  yarn (because both install `yarn` binaries)
/usr/local/Cellar/hadoop/hdfs (366 files, 40.9MB)
  Built from source
/usr/local/Cellar/hadoop/3.3.0 (21,844 files, 963.5MB) *

설정할 xml파일들은 아래 경로에 존재함

$ cd /usr/local/Cellar/hadoop/3.3.0/libexec/etc/hadoop

2. core-site.xml 설정

<configuration>
  <property>
     <name>hadoop.tmp.dir</name>
     <value>/usr/local/Cellar/hadoop/hdfs/tmp</value>
     <description>A base for other temporary directories.</description>
  </property>
   <property>
      <name>fs.defaultFS</name>
      <value>hdfs://localhost:9000</value>
   </property>
</configuration>

3. hdfs-site.xml 설정

<configuration>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

4. hdfs 실행

$ /usr/local/Cellar/hadoop/3.3.0/libexec/sbin/start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [pineappleui-iMac.local]
2020-08-19 16:54:52,420 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

 

 

추가...

참고로 맥을 껏다 켰을때 동작을 안하면 아래명령어로 네임노드를 포멧하고 테스트 재시작하면된다.

$ hadoop namenode -format