Erorr SparkTask

Four Stars

Erorr SparkTask

Hello everyone

 

I am using Talend Big Data Platform to implement ELT Jobs with Hive tables. The jobs run with Spark engine.

I have a problem with a Hive on Spark job.

The scope of the task is to join two tables (parquet) and then insert the output into another table (parquet).

The job goes wrong but I am not able to interpet it.

This is the error message:

 

[FATAL]: datahub_rp_cli.j_rp_cli_movimenti_0_1.J_RP_CLI_MOVIMENTI - tRunJob_2 Child job running failed
[statistics] disconnected
Exception in component tELTHiveOutput_9
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:279)

 

Trying to find the solution online in the official documentation or forum, I understand that the errors SparkTask (with code 1 or 2 or 3 eccetera) are something like a 'wrapper' of java errors; however I cannot access to hive log because of denied permissions on the server. 

So have you got a lists of causes that can originate somekind of java errors for Sparktask error code =3 ?

Thank you.

Moderator

Re: Erorr SparkTask

Hello,

Could you please make sure that all required external jars in your job are installed successfully in your studio? On which talend bigdata platform build version you got this issue?

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Four Stars

Re: Erorr SparkTask

Hi @xdshi 

I work with Talend Big Data Platform 6.3.1.

Where may I check if in Talend I have got all the right external Jars exactly installed?

However I have succeeded in resolving the problem in this way:

The Hive ELT process got error SparkTask at the table that is partictioned  daily from 2011 to 2018; I tested the Talend ELT job for a subset of this partitioned table and then it finished the execution well, without sparkTask error.

I imagine that I have to compute statistic for that table on hive, and so It is not a Talend software issue.

Thank you

 

Moderator

Re: Erorr SparkTask

Hello,

Great you could work on your solution.

Here is online documentation about:TalendHelpCenter:Installing external modules

Best regards

Sabrina

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now