Big data batch job failed while running from another standard/big data batch job

Four Stars

Big data batch job failed while running from another standard/big data batch job

Hello,

I am getting error while trying to run a big data batch job 'B' from another big data batch job 'A' or from a standard job using tRunJob.

 

But when I try to run Big data job 'B' independently, it is running fine.

Below is the code written in the job 'B' 

In tJava  - > 

 

SparkSession spark = SparkSession.builder().enableHiveSupport().config("hive.exec.dynamic.partition", "true").config("hive.exec.dynamic.partition.mode", "nonstrict").getOrCreate();

 

spark.sql("select cast(count(*) as string) from Database_1.missing_files_config WHERE data_source_name = 'source1' AND source_table_name = 'table1'").show();

 

And in tJava Advanced Settings -> 

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.Row;
import org.apache.spark.sql.Encoders;
import org.apache.hadoop.fs.LocatedFileStatus;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.RemoteIterator;
import org.apache.spark.sql.SaveMode;
import java.io.FileNotFoundException;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.HashMap;
import java.util.Map;
import org.apache.hadoop.fs.FSDataOutputStream;

 

Getting below error->

java.lang.ClassCastException: cannot assign instance of scala.collection.immutable.List$SerializationProxy to field
org.apache.spark.rdd.RDD.org$apache$spark$rdd$RDD$$dependencies_ of type scala.collection.Seq in instance of org.apache.spark.rdd.MapPartitionsRDD

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now

The Forrester Wave: Strategic iPaaS and Hybrid Integration Platforms, Q1 2019

Find out how Forrester rate different iPaaS and Hybrid Integration Platforms

Read