Hadoop Windows Native 编译说明

1、首先,下载hadoop-2.5.2-src源码

拷贝文件夹hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\native
拷贝文件夹from hadoop-2.5.2-src\hadoop-common-project\hadoop-common\src\main\winutils

2、设置JAVA_HOME及PATH环境变量

3、生成javah的头文件
解压hadoop-common-2.5.1.jar,然后运行

javah org.apache.hadoop.util.NativeCrc32
javah org.apache.hadoop.io.compress.lz4.Lz4Compressor
javah org.apache.hadoop.io.compress.lz4.Lz4Decompressor
javah org.apache.hadoop.io.nativeio.NativeIO
javah org.apache.hadoop.security.JniBasedUnixGroupsMapping
javah org.apache.hadoop.security.JniBasedUnixGroupsMapping

4、打开winutils.sln,修改输出路径到../bin,编译

5、打开native.sln,修改输出路径到../bin,修改winutils.lib引用地址,编译

6、拷贝exe及dll文件到HADOOP_HOME/bin,搞定

常见问题:
1、编译的硬件平台要与java位数一致(x86,x64),否则dll无法加载
2、出问题时,先运行winutils.exe,无法运行时,要安装对应VS版本的vcredist可再发行包就好了
3、如果提示”unable to load native hadoop-library for your platform”的话,那只需要在JVM启动参数中制定native library的路径,就可以了

如果比较着急的话,可以到我的github上下载2.5.2版本的native binary:hadoop-windows-native

Leave a Reply

Your email address will not be published. Required fields are marked *

*