hadoop윈도우에서 설치

hadoop

https://hadoop.apache.org/

 

github - 윈도우를 위한

https://github.com/ParixitOdedara/Hadoop

 

spark

http://spark.apache.org/

 

hadoop의 설정 파일 변경하기

- hadoop-env.cmd

- core-site.xml

- hdfs-site.xml

- mapred-site.xml

 

hadoop-env.cmd 필요정보

set HADOOP_IDENT_STRING=%USERNAME%
set HADOOP_PREFIX=%HADOOP_HOME%
set HADOOP_CONF_DIR=%HADOOP_PREFIX%\etc\hadoop
set YARN_CONF_DIR=%HADOOP_CONF_DIR%
set PATH=%PATH%;%HADOOP_PREFIX%\bin

 

core-site.xml

<configuration>
 <property>
     <name>fs.default.name</name>
     <value>hdfs://127.0.0.1:19000</value>
   </property> 
</configuration>

 

hdfs-site.xml

<configuration>
  <property>
      <name>dfs.replication</name>
      <value>1</value>
   </property>
   <property>
      <name>dfs.namenode.name.dir</name>
      <value>/C:/2019/bigdata/hadoop/hadoop-2.8.5/data/namenode</value>
   </property>
   <property>
      <name>dfs.datanode.data.dir</name>
      <value>/C:/2019/bigdata/hadoop/hadoop-2.8.5/data/datanode</value>
   </property>
</configuration>

 

mapred-site.xml

<configuration>
 <property>
      <name>mapreduce.job.user.name</name>
      <value>%USERNAME%</value>
   </property>
   <property>
      <name>mapreduce.framework.name</name>
      <value>yarn</value>
   </property>
   <property>
      <name>yarn.apps.stagingDir</name>
      <value>/user/%USERNAME%/staging</value>
   </property>
   <property>
      <name>mapreduce.jobtracker.address</name>
      <value>local</value>
   </property>
</configuration>