Reading a json external table into spark job issue

Five Stars

Reading a json external table into spark job issue

Hello,

 

I have a Spark Job with a Hive query  that reads an  external table (JSON format), but when i tried to get the colonne INPUT__FILE__NAME i have this error :

hive org.apache.spark.sql.AnalysisException: cannot resolve '`INPUT__FILE__NAME`'

 

My job

tHiveInput -> tMap -> tHiveOutput

 

How can i get the file name ?

 

My config:

Talend Big Data Batch Platform 6.5.1

Cloudera 5.12

 

 

Thanks

smer

 

Six Stars

Re: Reading a json external table into spark job issue

use the row name row1.filename

Five Stars

Re: Reading a json external table into spark job issue

Hello jcruie,

This solution in not working, because the row1.filename is unkown.
my job is
tHiveInput->(row1)->tmap->(row2)->thiveOutput
i use the row1.filename in ten tmap, and in the query hive but not working.

Regards
smer


Six Stars

Re: Reading a json external table into spark job issue

If filename is in tmap you need to use row2.filename