Big Data Spark Job - Process job based on file name

Six Stars

Big Data Spark Job - Process job based on file name

Hi,

 

I am creating big data spark job and have to check file name before proceeding to the specif workflow like

 

checkfilename -> (if contains "abc") -> then read file / process file

      |

(if filename contains "xyz") -> then read file/ diff process

 

OR

 

tInputfileDelimited -> Checkfilename (if contains "abc") -> then process file

                                     |

                                 (if filename contains "xyz") -> then diff process

 

Which component I have to use to implement above scenario. I tried to use tJavarow but runif has several limitation. We can connect tjavarow ->(runif)-> tJava only.

Moderator

Re: Big Data Spark Job - Process job based on file name

Hi,

In a standard job, here is a component tfileproperties which can create a single row flow that displays the properties of the processed file.

Best regards

Sabrina

 

 

 

 

 
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.
Five Stars

Re: Big Data Spark Job - Process job based on file name

I have similar requirement and tfileproperties is not available in spark job. Can you please suggest for a spark job. 

What’s New for Talend Spring ’19

Watch the recorded webinar!

Watch Now

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.

Download

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch