I am using Talend Big Data Platform to implement ELT Jobs with Hive tables. The jobs run with Spark engine.
I have a problem with a Hive on Spark job.
The scope of the task is to join two tables (parquet) and then insert the output into another table (parquet).
The job goes wrong but I am not able to interpet it.
This is the error message:
[FATAL]: datahub_rp_cli.j_rp_cli_movimenti_0_1.J_RP_CLI_MOVIMENTI - tRunJob_2 Child job running failed
Exception in component tELTHiveOutput_9
java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 3 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
Trying to find the solution online in the official documentation or forum, I understand that the errors SparkTask (with code 1 or 2 or 3 eccetera) are something like a 'wrapper' of java errors; however I cannot access to hive log because of denied permissions on the server.
So have you got a lists of causes that can originate somekind of java errors for Sparktask error code =3 ?
Could you please make sure that all required external jars in your job are installed successfully in your studio? On which talend bigdata platform build version you got this issue?
I work with Talend Big Data Platform 6.3.1.
Where may I check if in Talend I have got all the right external Jars exactly installed?
However I have succeeded in resolving the problem in this way:
The Hive ELT process got error SparkTask at the table that is partictioned daily from 2011 to 2018; I tested the Talend ELT job for a subset of this partitioned table and then it finished the execution well, without sparkTask error.
I imagine that I have to compute statistic for that table on hive, and so It is not a Talend software issue.
Great you could work on your solution.
Here is online documentation about:TalendHelpCenter:Installing external modules
Talend named a Leader.
Kickstart your first data integration and ETL projects.
Watch the recorded webinar!
Learn how to make your data more available, reduce costs and cut your build time
Read about OTTO's experiences with Big Data and Personalized Experiences
Take a look at this video about Talend Integration with Databricks