Six Stars

Big Data Spark Job - Process job based on file name

Hi,

 

I am creating big data spark job and have to check file name before proceeding to the specif workflow like

 

checkfilename -> (if contains "abc") -> then read file / process file

      |

(if filename contains "xyz") -> then read file/ diff process

 

OR

 

tInputfileDelimited -> Checkfilename (if contains "abc") -> then process file

                                     |

                                 (if filename contains "xyz") -> then diff process

 

Which component I have to use to implement above scenario. I tried to use tJavarow but runif has several limitation. We can connect tjavarow ->(runif)-> tJava only.

  • Big Data
  • Data Integration
  • Talend Integration Cloud
Tags (1)
1 REPLY
Moderator

Re: Big Data Spark Job - Process job based on file name

Hi,

In a standard job, here is a component tfileproperties which can create a single row flow that displays the properties of the processed file.

Best regards

Sabrina