Four Stars

Read S3 file in Talend Spark Job

error:

Checking ports...

Sending job 'Framework_Semantic_Abac' to server (10.1.6.23:8001)...

File transfer completed.

Deploying job 'Framework_Semantic_Abac' on server (10.1.6.23:8000)...

Running job 'Framework_Semantic_Abac'...
Starting job Framework_Semantic_Abac at 14:18 30/07/2018.

[statistics] connecting to socket on port 3954
[statistics] connected
[WARN ]: org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
[WARN ]: org.apache.spark.SparkConf - In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
[statistics] disconnected
[ERROR]: semantic_layer_se.framework_semantic_abac_0_1.Framework_Semantic_Abac - TalendJob: 'Framework_Semantic_Abac' - Failed with exit code: 1.
java.lang.IllegalArgumentException: Invalid hostname in URI s3n:

1 ACCEPTED SOLUTION

Accepted Solutions
Moderator

Re: Read S3 file in Talend Spark Job

Hello,

Did your secret key for S3 contains a "/" an invalid hostname?

Could you please refer to this issue page to see if it is what you are looking for?

https://issues.apache.org/jira/browse/HADOOP-3733?focusedCommentId=15132075&page=com.atlassian.jira....

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
1 REPLY
Moderator

Re: Read S3 file in Talend Spark Job

Hello,

Did your secret key for S3 contains a "/" an invalid hostname?

Could you please refer to this issue page to see if it is what you are looking for?

https://issues.apache.org/jira/browse/HADOOP-3733?focusedCommentId=15132075&page=com.atlassian.jira....

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.