Talend Spark job . Join using tmap Issue

Five Stars

Talend Spark job . Join using tmap Issue

Hi,

 

I have created a simple BD spark job. I have used tmap where I am looking up to a few files.

The compilation is fine, The lookup hdfs files have contents in them.

But the Code fails during execution: Error message below.

If any one has faced similar issue, please let me know how was it resolved.

 

 

org.talend.bigdata.dataflow.SpecException: Invalid input accessor: clsn_62.null
at org.talend.bigdata.dataflow.hmap.HMapSpec$JoinDef.deserialize(HMapSpec.java:1206)
at org.talend.bigdata.dataflow.hmap.HMapSpec$JoinDef.access$1800(HMapSpec.java:1122)
at org.talend.bigdata.dataflow.hmap.HMapSpec.joinKey(HMapSpec.java:563)
at org.talend.bigdata.dataflow.hmap.HMapSpecBuilder.joinKey(HMapSpecBuilder.java:171)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.tHiveInput_2Process(tdata_wh_BD_spec.java:2557)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.tFileInputDelimited_6_HDFSInputFormatProcess(tdata_wh_BD_spec.java:4458)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.run(tdata_wh_BD_spec.java:4856)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.runJobInTOS(tdata_wh_BD_spec.java:4672)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.main(tdata_wh_BD_spec.java:4554)
org.talend.bigdata.dataflow.SpecException: Invalid input accessor: clsn_62.null
at org.talend.bigdata.dataflow.hmap.HMapSpec$JoinDef.deserialize(HMapSpec.java:1206)
at org.talend.bigdata.dataflow.hmap.HMapSpec$JoinDef.access$1800(HMapSpec.java:1122)
at org.talend.bigdata.dataflow.hmap.HMapSpec.joinKey(HMapSpec.java:563)
at org.talend.bigdata.dataflow.hmap.HMapSpecBuilder.joinKey(HMapSpecBuilder.java:171)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.tHiveInput_2Process(tdata_wh_BD_spec.java:2557)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.tFileInputDelimited_6_HDFSInputFormatProcess(tdata_wh_BD_spec.java:4458)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.run(tdata_wh_BD_spec.java:4856)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.runJobInTOS(tdata_wh_BD_spec.java:4672)
at t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec.main(tdata_wh_BD_spec.java:4554)
[ERROR]: t_data_wh.tdata_wh_bd_spec_0_1.tdata_wh_BD_spec - TalendJob: 'tdata_wh_BD_spec' - Failed with exit code: 1.


Accepted Solutions
Forteen Stars

Re: Talend Spark job . Join using tmap Issue

@badri-nair ,check below link,

 

1- Make sure the input for this flow ( row1 ) here 0 is not empty, this input can be a source file or a db query

2- make sure you are not joining on multiple keys, if you need to do so , please one tmap for each key join

http://talendexpert.com/talend-spark-error-2/

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.

All Replies
Forteen Stars

Re: Talend Spark job . Join using tmap Issue

@badri-nair ,check below link,

 

1- Make sure the input for this flow ( row1 ) here 0 is not empty, this input can be a source file or a db query

2- make sure you are not joining on multiple keys, if you need to do so , please one tmap for each key join

http://talendexpert.com/talend-spark-error-2/

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Five Stars

Re: Talend Spark job . Join using tmap Issue

Thank you very much, Manohar. 

had to use 4 tmaps for 4 joins. I had in just one tmap in the previous standard job. But it does work.

 

 

Thanks

Badri Nair 

 

Forteen Stars

Re: Talend Spark job . Join using tmap Issue

@badri-nair ,can you show your job design and you can first load to tHashoutput and read that using tHashInput ,if you want to take multiple lockups as you can use multiple tHasInputs

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Five Stars

Re: Talend Spark job . Join using tmap Issue

HI Manohar, 

there is no thash components for a spark job. only tCache .

In the standard job i had 4 thash outputs and 4thash inputs was linked to each one of them. They 4 inputs were used as lookup in a single tmap.

Looks like that cant be done inBD-job.JPGSTD-job.JPG a spark job .

 

Thanks

Badri Nair 

 

 

 

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now