Error 'GSSException: No valid credentials provided' while connecting to Kerberos and SSL-enabled Impala

Talend Version          6.1.1

Summary

 
Additional Versions 5.6.1, 5.6.2, 6.0.1, 6.1.1
Key words javax.security.sasl.SaslException ,GSS initiate failed ,Message stream modified (41)
Product Talend Big Data
Component Studio
Article Type Debugging
Problem Description

The error shown below appears when connecting Studio to a Kerberized Hadoop Cluster:

javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Message stream modified (41))]
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
	at org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:413)
	at org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:558)
	at org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:373)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:727)
	at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:723)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
	at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:722)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
	at org.apache.hadoop.ipc.Client.call(Client.java:1397)
	at org.apache.hadoop.ipc.Client.call(Client.java:1358)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy7.create(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:373)
	at org.apache.hadoop.ipc.Client.getConnection(Client.java:1493)
	at org.apache.hadoop.ipc.Client.call(Client.java:1397)
	at org.apache.hadoop.ipc.Client.call(Client.java:1358)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
	at com.sun.proxy.$Proxy7.create(Unknown Source)
	at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy8.create(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
	at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
	at com.sun.proxy.$Proxy8.create(Unknown Source)
	at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1692)
	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
	at org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1692)
	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
	at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
	at org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
	at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:444)
	at org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.tFileInputDelimited_1Process(Put_file_to_hadoop.java:1826)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.tRowGenerator_1Process(Put_file_to_hadoop.java:1457)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:909)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:890)
	at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:787)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.tFileInputDelimited_1Process(Put_file_to_hadoop.java:1826)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.tRowGenerator_1Process(Put_file_to_hadoop.java:1457)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.tHDFSConnection_1Process(Put_file_to_hadoop.java:813)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.runJobInTOS(Put_file_to_hadoop.java:5819)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.main(Put_file_to_hadoop.java:5573)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.tHDFSConnection_1Process(Put_file_to_hadoop.java:813)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.runJobInTOS(Put_file_to_hadoop.java:5819)
	at talend_ivp.put_file_to_hadoop_0_1.Put_file_to_hadoop.main(Put_file_to_hadoop.java:5573)
Caused by: GSSException: No valid credentials provided (Mechanism level: Message stream modified (41))
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:770)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:248)
	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)
	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)
	... 39 more
Caused by: KrbException: Message stream modified (41)
	at sun.security.krb5.KrbKdcRep.check(KrbKdcRep.java:50)
	at sun.security.krb5.KrbTgsRep.<init>(KrbTgsRep.java:87)
	at sun.security.krb5.KrbTgsReq.getReply(KrbTgsReq.java:259)
	at sun.security.krb5.KrbTgsReq.sendAndGetCreds(KrbTgsReq.java:270)
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:302)
	at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:120)
	at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:458)
	at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:693)
Problem root cause
  1. The JVM spun by Studio is not able to find the krb5.ini file on Windows.
  2. Multiple Javas are installed on the Windows machine.

    1. If you have two Javas on the same machine, then you go to a command prompt and type C:/Java_6/jre/bin/ klist, you would see some tickets.
    2. If you type C:/java_7/jre/bin/klist, it would show you different tickets.
Solution or Workaround
  1. Start up Studio using Java 1.8 or Java 1.7, whichever is compatible with your Talend version.
  2. Navigate to Advance Settings of the Run tab view of the Talend Job, and set:

    -Djava.security.krb5.conf =C:/ProgramData/MIT/Kerberos/krb5.ini

    Be sure to avoid any space in ProgramData.

    VMarg.png

    put_file_to_hadoop.png

  3. Also set the following for debug:

    -Djavax.net.debug=ssl
JIRA ticket number  
Version history
Revision #:
9 of 9
Last update:
‎09-29-2018 12:17 AM
Updated by: