tHDFSInput error NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V

[pcaldeira]

Please, someone can help me?

I´m developing a job that needs to connect to Azure Data Factory e and connect a Hadoop database.

I´m using Talend Big Data version TOS_BD-20170623_1246-V6.4.1 and I´m importing these libraries:

• azure-data-lake-store-sdk-2.3.6.jar
• hadoop-azure-datalake-3.2.0.jar
• jackson-core-2.8.4.jar

And I´m following the steps of this link https://help.talend.com/reader/Sm466hmdh~Y~2GehtIo6xw/lzTTE9~8uYW46A4QuKvJKA

I guess that problem is at tHDFSInput.

And I´m receiving the error bellow:

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V
at org.apache.hadoop.fs.adl.AdlConfKeys.addDeprecatedKeys(AdlConfKeys.java:127)
at org.apache.hadoop.fs.adl.AdlFileSystem.<clinit>(AdlFileSystem.java:98)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Unknown Source)
at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2702)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2715)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2751)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2733)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:377)
[statistics] disconnected
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:166)
at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Unknown Source)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)
at local_project.connection_azure_0_1.connection_azure.tHDFSInput_1Process(connection_azure.java:824)
at local_project.connection_azure_0_1.connection_azure.runJobInTOS(connection_azure.java:1256)
at local_project.connection_azure_0_1.connection_azure.main(connection_azure.java:1093)

[xdshi]

Hello,

What's the version of hadoop you are using and what's your HD Insight? Make sure that the Azure JARs you are using has the correct version of hadoop-common using.

Best regards

Sabrina

[damaral]

We are using the “Hadoop 3.1 (HDI 4.0)” version and the JARs have been configured correctly.
After execution we had the error “org.apache.hadoop.util.Shell - Failed to locate binary winutils in hadoop binary path”, what action can we do to fix this error?

[pcaldeira]

Hi, damaral.

Please follow the steps described at link  https://community.talend.com/t5/Design-and-Development/resolved-Winutils-library/m-p/99377#M57888 to solve the problem about “org.apache.hadoop.util.Shell - Failed to locate binary winutils in hadoop binary path”.

But I´m still suffering the error "tHDFSInput error NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V" in my job.

Best regards.

pcaldeira

[damaral]

Thank you pcaldeira!

The changes / settings have been made and now I am having the following error:

Starting job Jtesteconexaohadoop at 16:22 28/08/2019.

[statistics] connecting to socket on port 3724

[statistics] connected

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/D:/TOS_BD-20190620_1446-V7.2.1/TOS_BD-20190620_1446-V7.2.1/configuration/.m2/repository/org/talend/libraries/slf4j-log4j12-1.7.10/6.0.0/slf4j-log4j12-1.7.10-6.0.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/D:/TOS_BD-20190620_1446-V7.2.1/TOS_BD-20190620_1446-V7.2.1/configuration/.m2/repository/org/slf4j/slf4j-log4j12/1.7.16/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

Exception in thread "main" java.lang.NoSuchMethodError: org.apache.hadoop.conf.Configuration.reloadExistingConfigurations()V

      at org.apache.hadoop.fs.adl.AdlConfKeys.addDeprecatedKeys(AdlConfKeys.java:114)

      at org.apache.hadoop.fs.adl.AdlFileSystem.<clinit>(AdlFileSystem.java:92)

      at java.lang.Class.forName0(Native Method)

      at java.lang.Class.forName(Unknown Source)

      at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)

      at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)

      at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)

      at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2702)

      at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2715)

      at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:93)

      at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2751)

      at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2733)

      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:377)

      at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:166)

      at org.apache.hadoop.fs.FileSystem$1.run(FileSystem.java:163)

      at java.security.AccessController.doPrivileged(Native Method)

      at javax.security.auth.Subject.doAs(Unknown Source)

      at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698)

      at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:163)

      at local_project.jtesteconexaohadoop_0_1.Jtesteconexaohadoop.tFixedFlowInput_1Process(Jtesteconexaohadoop.java:584)

      at local_project.jtesteconexaohadoop_0_1.Jtesteconexaohadoop.runJobInTOS(Jtesteconexaohadoop.java:1818)

      at local_project.jtesteconexaohadoop_0_1.Jtesteconexaohadoop.main(Jtesteconexaohadoop.java:1668)

[statistics] disconnected

Job Jtesteconexaohadoop ended at 16:22 28/08/2019. [exit code=1]