I am creating big data spark job and have to check file name before proceeding to the specif workflow like
checkfilename -> (if contains "abc") -> then read file / process file
(if filename contains "xyz") -> then read file/ diff process
tInputfileDelimited -> Checkfilename (if contains "abc") -> then process file
(if filename contains "xyz") -> then diff process
Which component I have to use to implement above scenario. I tried to use tJavarow but runif has several limitation. We can connect tjavarow ->(runif)-> tJava only.
In a standard job, here is a component tfileproperties which can create a single row flow that displays the properties of the processed file.
I have similar requirement and tfileproperties is not available in spark job. Can you please suggest for a spark job.
Watch the recorded webinar!
Accelerate your data lake projects with an agile approach
Create systems and workflow to manage clean data ingestion and data transformation.
Introduction to Talend Open Studio for Data Integration.