HiBench compatibility issues with Hadoop 3.x in micro.dfsioe prepare part...

[ZeweiChen11]

We met the error when preparing micro.dfsioe on CDH 6.0 where hadoop version is 3.0.

18/05/07 21:03:25 INFO dfsioe.TestDFSIOEnh: nrFiles = 2048 18/05/07 21:03:25 INFO dfsioe.TestDFSIOEnh: fileSize (MB) = 1000 18/05/07 21:03:25 INFO dfsioe.TestDFSIOEnh: bufferSize = 4096 Exception in thread "main" java.lang.IllegalAccessError: class org.apache.hadoop.hdfs.web.HftpFileSystem cannot access its superinterface org.apache.hadoop.hdfs.web.TokenAspect$TokenManagementDelegator at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:760) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362)

So we rebuilded HiBench-7.0 using command
mvn -Phadoopbench -Psparkbench -Dhadoop.mr2.version=3.0.0 -Dspark=2.2 -Dscala=2.11 clean package,
it reports error
[ERROR] /root/HiBench_7.0/autogen/src/main/java/org/apache/hadoop/fs/dfsioe/TestDFSIO.java:[73,49] incompatible types: org.slf4j.Logger cannot be converted to org.apache.commons.logging.Log

[LifengWang]

There is another issue when building Hibench on Hadoop 3.0.0
It seems that copyMerge function in FileUtil.java doesn't exist anymore in Hadoop 3.0

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.2:compile (default-compile) on project autogen: Compilation failure [ERROR] /home/alex/workspace/HiBench/autogen/src/main/java/org/apache/hadoop/fs/dfsioe/TestDFSIOEnh.java:[955,34] 找不到符号 [ERROR] 符号: 方法 copyMerge(org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path,org.apache.hadoop.fs.FileSystem,org.apache.hadoop.fs.Path,boolean,org.apache.hadoop.conf.Configuration,<nulltype>) [ERROR] 位置: 类 org.apache.hadoop.fs.FileUtil

[gcz2022]

Sorry but we currently don't support Hadoop 3.0

[technmsg]

Are there any plans to support Hadoop 3.0?

[bfarsi]

Is there any update on supporting Hadoop 3.0 ?

[hexiangheng]

Is there any update on supporting Hadoop 3.0 ?

[godtree]

Is there any update on supporting Hadoop 3.0 ?

[gcz2022]

@technmsg @bfarsi @hexiangheng @godtree See #586 Hadoop 3.2 is going to be supported after this is merged. I suppose Hadoop 3.0 shouldn't require too many code changes.

[godtree]

@gczsjdy I am looking forward to it. If it works, tell me please. Thanks.

[godtree]

@gczsjdy And will CDH 6.x be supported after the next merged?