Error while running a spark job on AWS EMR.

Four Stars

Error while running a spark job on AWS EMR.

I have a job that is running well in local mode. But when I try to launch the same job on AWS EMR, I get the following error:  This is the error as seen from the Spark log files.

Diagnostics Info:
AM Container for appattempt_1525124598990_0007_000002 exited with exitCode: -1000
Failing this attempt.Diagnostics: File file:/Users/Desktop/TALEND/Talend-Studio-20180116_1512-V6.5.1/workspace/.Java/lib/metrics-core-3.1.2.jar does not exist
java.io.FileNotFoundException: File file:/Users/Desktop/TALEND/Talend-Studio-20180116_1512-V6.5.1/workspace/.Java/lib/metrics-core-3.1.2.jar does not exist
at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:640)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:866)
at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:630)
at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:452)
at org.apache.hadoop.yarn.util.FSDownload.copy(FSDownload.java:253)
at org.apache.hadoop.yarn.util.FSDownload.access$000(FSDownload.java:63)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:361)
at org.apache.hadoop.yarn.util.FSDownload$2.run(FSDownload.java:359)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:359)
at org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
For more detailed output, check the application tracking page: http://ip-10-0-0-12.ec2.internal:8088/cluster/app/application_1525124598990_0007 Then click on links to logs of each attempt.

 

Note:
Looks like the talend job is trying to find the file "metrics-core-3.1.2.jar" from my local(developer) machine, but it is unable to find it. However I verified that the file exists and is accessible.

 

Can anyone help ?

Moderator

Re: Error while running a spark job on AWS EMR.

Hello,

The metrics-core-3.1.2.jar file should be shipped within Talend Studio. Could you please create a case on talend support portal so that we can give you a remote assistance(webex) through support portal with priority?

 

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
One Star

Re: Error while running a spark job on AWS EMR.

Did you resolve your issue? If so, can you please share your solution? Currently, I am experiencing the same problem.

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download