ã1.å¨Linuxç¯å¢ä¸ä¼ªåå¸å¼é¨ç½²hadoop(SSHå
ç»é)ï¼è¿è¡WordCountå®ä¾æåã
http://www.cnblogs.com/PurpleDream/p/4009070.html 2.èªå·±æå
hadoopå¨eclipseä¸çæ件ã
http://www.cnblogs.com/PurpleDream/p/4014751.html 3.å¨eclipseä¸è®¿é®hadoopè¿è¡WordCountæåã
æ以æä¸è¾¹ä¼åä¸æ¬¡è®°å½ä¸æçè¿ç¨ï¼ä¸ºèªå·±ä»¥åæ¥é
æ¹ä¾¿ï¼è¦æ¯è½å¸®å©å°å
¶ä»äººï¼èªç¶æ¯æ´å¥½äºï¼
===============================================================é¿é¿çåå²çº¿====================================================================
æ£æ:
å¦æä½ æ³å¨eclipseä¸å®è£
hadoopçæ件ï¼ä¸ä¸ªå¾éè¦çé®é¢å°±æ¯hadoopççæ¬ä¸eclipseçæ¬çå¹é
é®é¢ï¼å¦æä¸å¹é
ï¼å¯è½ä¼å¯¼è´å¾å¤é®é¢ã
综ä¸ï¼æå¨å®æäºå¨LinuxçCentOSç³»ç»ä¸å®è£
hadoop并ç®åè¿è¡WordCountä¹å(å
·ä½åçæ¬æåè¨ä¸çå客é¾æ¥)ï¼å°å¦ä¹ çéç¹æ¾å°äºhadoopçeclipseæ件ä¸ãå
¶å®ç½ä¸ä¹æé¨åæç« å¨ä»ç»å¦ä½ç¼åæ件ï¼ä½æ¯ç±äºæçeclispeçæ¬æ¯Junoï¼èä¹åå¨CentOSå®è£
çæ¯hadoop-1.0.1ï¼æ以没ææ¾å°å®å
¨å¹é
çæåæ¡ä¾ãæ以æç»å³å®èªå·±ä¹å¨æåå¼ä¸ä¸ªèªå·±çhadoopæ件ã
å¨æäºè¿ä¸ªæ³æ³ä¹åï¼ä¸å¼å§ä¼è§å¾ç¹å«å°é¾ï¼ä½æ¯å¨çæ£å»åä¹åï¼ä¼åç°å¾å¤é®é¢å
¶å®è¿æ¯å¯ä»¥è§£å³çï¼åªè¦ä½ æå¾å¦æ使ç¨ç¾åº¦åè°·æï¼å¤åèä¸ä¸å«äººçæç« ï¼æ±²åç²¾åï¼æç»ä¸å®ä¼æåçã
第ä¸æ¥ï¼ç¡®å®å¤§è´æµç¨:
ãã 1.é¦å
æéè¦antç¼è¯ï¼ç¶åå°±æ¯hadoop-1.0.1.tar.gzè¿ä¸ªå®è£
å
ï¼è¿æå°±æ¯eclipseã
2.é对æèªå·±çç¯å¢ï¼çµèæ¯Win7ï¼eclispeæ¯Junoï¼ä¸è½½äºhadoop-1.0.1.tar.gzï¼å°±å·®antä¹å没æå®è£
ã
第äºæ¥ï¼å®è£
antï¼
ãã 1.æåèçæ¯è¿ç¯æç«
http://blog.csdn.net/yang382197207/article/details/10185251ï¼æå½æ¶ä¸è½½çæ¯apache-ant-1.9.4ï¼å
¶ä»çæ¥éª¤æç
§è¿ç¯æç« ä»ç»çé
置就ä¼æåã
第ä¸æ¥ï¼å¨æ£å¼å»ºç«é¡¹ç®ä¹åï¼æå¨ä»ç»ä¸æçç¯å¢ï¼ OS: windows 7, Eclipse: Juno, JDK: 1.6.0_43, Hadoop: 1.0.1
1.é¦å
å¨Eclipseä¸æ°å»ºä¸ä¸ªJava项ç®ï¼ååå«hadoop-1.0.1-eclipse-pluginï¼å
¶å®è¿ä¸ªååä½ å¯ä»¥éæçã
2.解åhadoop-1.0.1.tar.gzï¼å¨è§£ååçç®å½ä¸(æçæ¯D:\SettingUp\ITSettingUp\Hadoop\hadoop-1.0\hadoop-1.0.1)ï¼ä¾æ¬¡å°ä¸é¢çæ件æ¾å°é¡¹ç®ä¸:
(1).å°\hadoop-1.0.1\src\contrib\eclipse-pluginéé¢çæææ件以å\hadoop-1.0.1\src\contrib\build-contrib.xmlææ½å°é¡¹ç®ä¸
(2).å°æ ¹ç®å½\hadoop-1.0.1éçææ.jaræ件ææ½å°eclipse项ç®ä¸binç®å½ä¸ã
3.æå¼é¡¹ç®ä¸çbuild-contrib.xmlï¼åå¦ä¸çé
ç½®ï¼
(1).æ¾å°<property name="hadoop.root" location="hadoop-1.0.1解å缩åçç®å½"/>
(2).æ¾å°<property name="eclipse.home" location="èªå·±çeclipseçç®å½"/>
(3).æ¾å°<property name="version" value="hadoopççæ¬å·"/>
å¯ä»¥åèæçé
ç½®ï¼å¦å¾:
view sourceprint?
1.
<property name="name" value="${ant.project.name}"/>
2.
<property name="root" value="${basedir}"/>
3.
<property name="hadoop.root" location="D:/SettingUp/ITSettingUp/Hadoop/hadoop-1.0/hadoop-1.0.1"/>
4.
<property name="eclipse.home" location="D:/HaveSetted/ITHaveSetted/Eclipse"/>
5.
<property name="version" value="1.0.1"/>
4.æå¼é¡¹ç®ä¸çbuild.xmlï¼åå¦ä¸çé
ç½®ï¼
(1).å°æ件å¼å¤´ç <import file="../build-contrib.xml"/> ä¿®æ¹ä¸º <import file="build-contrib.xml"/>
(2).å¨<javac...>éå¦ä¸å¾å å
¥includeantruntime="on"ï¼ææå¦ä¸ï¼
view sourceprint?
01.
<target name="compile" depends="init, ivy-retrieve-common" unless="skip.contrib">
02.
<echo message="contrib: ${name}"/>
03.
<javac
04.
encoding="${build.encoding}"
05.
srcdir="${src.dir}"
06.
includes="**/*.java"
07.
destdir="${build.classes}"
08.
debug="${javac.debug}"
09.
deprecation="${javac.deprecation}"
10.
includeantruntime="on">
11.
<classpath refid="classpath"/>
12.
</javac>
13.
</target>
(3).å¨<path id="classpath">ä¸æ·»å ï¼<path refid="hadoop-jars"/>ï¼ææå¦ä¸ï¼
view sourceprint?
1.
<!-- Override classpath to include Eclipse SDK jars -->
2.
<path id="classpath">
3.
<pathelement location="${build.classes}"/>
4.
<pathelement location="${hadoop.root}/build/classes"/>
5.
<path refid="eclipse-sdk-jars"/>
6.
<path refid="hadoop-jars"/>
7.
</path>
(4).å¨<target name="jar" depends="compile" unless="skip.contrib">è¿ä¸ªæ ç¾éï¼å¨<copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>è¿ä¸ªé
ç½®çä¸é¢æ·»å å¦ä¸å 个jarå
çé
ç½®ï¼
view sourceprint?
1.
<copy file="${hadoop.root}/lib/commons-configuration-1.6.jar" todir="${build.dir}/lib" verbose="true"/>
2.
<copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar" todir="${build.dir}/lib" verbose="true"/>
3.
<copy file="${hadoop.root}/lib/commons-lang-2.4.jar" todir="${build.dir}/lib" verbose="true"/>
4.
<copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar" todir="${build.dir}/lib" verbose="true"/>
5.
<copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar" todir="${build.dir}/lib" verbose="true"/>
æç»ææå¦å¾ï¼
view sourceprint?
01.
<!-- Override jar target to specify manifest -->
02.
<target name="jar" depends="compile" unless="skip.contrib">
03.
<mkdir dir="${build.dir}/lib"/>
04.
<copy file="${hadoop.root}/build/hadoop-core-${version}.jar" tofile="${build.dir}/lib/hadoop-core-1.0.1.jar" verbose="true"/>
05.
<copy file="${hadoop.root}/build/ivy/lib/Hadoop/common/commons-cli-${commons-cli.version}.jar" todir="${build.dir}/lib" verbose="true"/>
06.
07.
<copy file="${hadoop.root}/lib/commons-configuration-1.6.jar" todir="${build.dir}/lib" verbose="true"/>
08.
<copy file="${hadoop.root}/lib/commons-httpclient-3.0.1.jar" todir="${build.dir}/lib" verbose="true"/>
09.
<copy file="${hadoop.root}/lib/commons-lang-2.4.jar" todir="${build.dir}/lib" verbose="true"/>
10.
<copy file="${hadoop.root}/lib/jackson-core-asl-1.8.8.jar" todir="${build.dir}/lib" verbose="true"/>
11.
<copy file="${hadoop.root}/lib/jackson-mapper-asl-1.8.8.jar" todir="${build.dir}/lib" verbose="true"/>
12.
13.
<jar
14.
jarfile="${build.dir}/hadoop-${name}-${version}.jar"
15.
manifest="${root}/META-INF/MANIFEST.MF">
16.
<fileset dir="${build.dir}" includes="classes/ lib/"/>
17.
<fileset dir="${root}" includes="resources/ plugin.xml"/>
18.
</jar>
19.
</target>
(5).å¨æ件æ«å°¾</project>æ ç¾ä¹åæ·»å é
ç½®ï¼
view sourceprint?
1.
<path id="hadoop-jars">
2.
<fileset dir="${hadoop.root}/">
3.
<include name="hadoop-*.jar"/>
4.
</fileset>
5.
</path>
5.æå¼hadoop-1.0.1ç解å缩ç®å½(æçæ¯D:\SettingUp\ITSettingUp\Hadoop\hadoop-1.0\hadoop-1.0.1)ï¼å¤å¶ä¸¤ä¸ªjarå
å°å¯¹åºçç®å½æ件夹ï¼ä»¥æçæ件ç®å½ç»æ举ä¾å¦ä¸ï¼
(1).å°D:\SettingUp\ITSettingUp\Hadoop\hadoop-1.0\hadoop-1.0.1\hadoop-core-1.0.1.jar å¤å¶å°D:\SettingUp\ITSettingUp\Hadoop\hadoop-1.0\hadoop-1.0.1\build\è¿ä¸ªæ件夹ä¸
(2).å°D:\SettingUp\ITSettingUp\Hadoop\hadoop-1.0\hadoop-1.0.1\lib\commons-cli-1.2.jarå¤å¶å°D:\SettingUp\ITSettingUp\Hadoop\hadoop-1.0\hadoop-1.0.1\build\ivy\lib\Hadoop\commonè¿ä¸ªæ件夹ä¸
6.ååå°eclipseï¼æå¼é¡¹ç®ä¸ç\META-INF\MANIFEST.MFæ件ï¼åå¦ä¸çæ·»å ï¼
view sourceprint?
1.
Bundle-ClassPath: classes/,
2.
lib/hadoop-core-1.0.1.jar,
3.
lib/commons-cli-1.2.jar,
4.
lib/commons-configuration-1.6.jar,
5.
lib/commons-httpclient-3.0.1.jar,
6.
lib/commons-lang-2.4.jar,
7.
lib/jackson-core-asl-1.8.8.jar,
8.
lib/jackson-mapper-asl-1.8.8.ja,
7.ç»è¿ä¸è¾¹çé
ç½®ï¼åºæ¬å·²ç»å®æï¼æ们å¯ä»¥ç¼è¯é¡¹ç®äºãå³å»build.xmlæ件éæ© Run As Ant Buildï¼ç¼è¯æåï¼çæçhadoop-1.0.1-eclipse-plugin.jarå¨hadoop-1.0.1解åç®å½ç\build\contrib\eclipse-pluginæ件夹éé¢ã
第åæ¥ï¼å½ç¶å°±æ¯å¨eclipseä¸é
ç½®æ们ç¼è¯å¥½çæ件ã
1.æhadoop-1.0.1-eclipse-plugin.jaræ件æ¾å°eclipseå®è£
ç®å½çpluginsä¸ï¼éå¯eclipseãéå¯åï¼ä¼å¨Window->Open Perspective->Otheréï¼çå°å¤§è±¡å¾æ çMap/Reduceåºç°ï¼æè
æ¯window->Preferencesä¸çå°Hadoop Map/Reduceè¿ä¸ªé项ã
2.window->Preferencesä¸ç¹å»Hadoop Map/Reduceè¿ä¸ªé项ï¼å°Hadoopçé£ä¸ªè§£å缩ç®å½(æçæ¯D:\SettingUp\ITSettingUp\Hadoop\hadoop-1.0\hadoop-1.0.1)çè·¯å¾é
ç½®ä¸ï¼é
ç½®ææå¾å¦ä¸ï¼
\
3.ç¶åæ们å¨Window->Open Perspective->Otheréï¼ç¹å»Map/Reduce,åæ¢å°è¿ä¸ªæ¨¡å¼ï¼æªå¾å¦ä¸ï¼
\
4.åæ¢å°Map/Reduce模å¼åï¼æ们çå°ç»é¢(4)ä¸çæªå¾ï¼ç¹å»å³ä¸è§ç两个å°å¾æ ï¼æµè¯æ没æååºï¼å¦ææååºï¼ææ¶å
å«é
ç½®ï¼è¿ä¸ªæä¸ç¯å客ä¼å
·ä½è®²ãå¦ææä¹ç¹é½æ²¡æååºï¼åå¯ä»¥åå¦ä¸æä½ï¼
(1).æå¼é¡¹ç®ä¸ç\META-INF\MANIFEST.MFæ件ï¼æ¾å°ä¹åç¼è¯æ¶ï¼æ·»å çBundle-ClassPath: classes/ è¿ä¸ªä½ç½®ï¼ä¹åæ们添å äº7个jarå
çé
ç½®ï¼åå¢å å个ï¼å¦ä¸ï¼
view sourceprint?
1.
lib/hadoop-common-0.21.0.jar,
2.
lib/hadoop-hdfs-0.21.0.jar,
3.
lib/log4j-1.2.15.jar,
4.
lib/hadoop-mapred-0.21.0.jar
(2).ç¶åæ们è¿éè¦å»ç½ä¸ä¸è½½è¿å个jarå
æ¾å°ä¸é¢çæ件夹ä¸ï¼ä»¥æçç®å½ä¸¾ä¾ï¼D:\SettingUp\ITSettingUp\Hadoop\hadoop-1.0\hadoop-1.0.1\libã
(3).ç¶åå¨éæ°ç¼è¯å³å¯ï¼å¨æ¿æ¢åæ¥çæ件æ¶ï¼å¨éå¯eclipseçè¿ç¨ä¸ï¼ä½ å¯ä»¥è¿æ ·åï¼å
å é¤åæ¥çæ件ï¼åæ¥éå¯eclipseï¼ç¡®è®¤Map/Reduce没äºï¼ç¶ååå
³æeclipseï¼å°æ°çæ件æ¾å°pluginséï¼ä¹åå次å¯å¨eclipseã解å³è¿ä¸ªé®é¢æ¶æå¨ç½ä¸åèçæç« æ¯ï¼
http://blog.csdn.net/kky2010_110/article/details/7774747ï¼å¤§å®¶ä¹å¯ä»¥çç~ã
(4).ä¸è¾¹ææå°çæªå¾å¦ä¸ï¼
\
ç»è¿ä¸è¾¹è¯´çè¿äºæ¥éª¤ï¼eclipseä¸çhadoopæ件æ们ç®æ¯ç¼è¯å®äºï¼è¿ä¸ªè¿ç¨ä¸ä¸å®è¦æ³¨æçæ¯ï¼ä¸è¦æ失败ï¼å¤å°è¯ï¼å 为hadoopçæ¬ä¸eclipseçæ¬æ¯æå
³ç³»çï¼æ以导è´ä½ ççæ¬åæçä¸å¤ªä¸è´ï¼å¯è½ä¼å¤±è´¥ï¼ä½æ¯ä¹å¯è½ä¼æåãæ以å¨è¿éå次æéï¼æ³¨ææç« å¼å§ï¼æ对èªå·±çç¯å¢ççæ¬çä»ç»ï¼å°¤å
¶æ¯eclipseåhadoopççæ¬ã建议大家å¦æææ¶é´çè¯ï¼å¯ä»¥å
ä¸ç®¡çæ¬ï¼æç
§æçæ¥éª¤å
èµ°ä¸éï¼å¦æå®å¨æ¯ä¸æåçè¯ï¼åå¯ä»¥æ¾ä¸ªåæä¸æ ·çeclipseåhadoopçæ¬ï¼åä½éªä¸åãå 为ï¼æå¨è¿ä¸ªè¿ç¨ä¸ï¼å°±åç°ï¼ç±äºæ¯ç¬¬ä¸æ¬¡å¦hadoopï¼ä¸è§å°æåçææï¼æ»è§å¾å°ç¹ä»ä¹ï¼æ»æ³è§è¯ä¸ä¸ï¼è«åç¬å~
è³äºä¸ä¸æ¥ï¼å¦æé
ç½®Locationåè¿è¡WordCountï¼æä¼å¨ä¸ä¸ç¯å客ä¸å
·ä½è¯´æï¼è°¢è°¢æ¯æï¼è§å¾æç« è¿ä¸éçè¯ï¼å¤å¤ç¹èµï¼å¤å¤çè¨åï¼è¿å°ä½¿æå
满å¨å!