check for specific and latest file, if found then process files one by one

Five Stars

check for specific and latest file, if found then process files one by one

I am getting a number of files like below in incoming folder 

file_from_US_3_june.csv

file_from_IND_12_may.csv

file_from_US_4_jan.csv 

file_from_UK_15_may.csv

file_from_IND_10_jan.csv

file_from_FR_9_apr.csv

 

if (check US file && IND file)

        get the latest modified US file and IND file 

        iterate and process files 

else

       no process

 

How can i execute the above in Talend 

 

Note - I want to check only US and IND files , if both files are present then only get the latest file of US and IND.... Process the latest file further 

Community Manager

Re: check for specific and latest file, if found then process files one by one

Use a tFileList and "Order by" modified date. Then check the filename supplied by the ((String)globalMap.get("tFileList_#_CURRENT_FILE")) globalMap variable to see if it is worth processing. If it is, use the filepath variable ((String)globalMap.get("tFileList_1_CURRENT_FILEPATH")) to point your file component to it. 

 

I think you are confusing the issue by combining the country and date filtering/ordering. You can simply process the files in order (assuming all will be processed) and selectively process them according to the IND/US section in the name.

Five Stars

Re: check for specific and latest file, if found then process files one by one

I can only process if US and IND files are present
Five Stars

Re: check for specific and latest file, if found then process files one by one

Thanks i will try this. 

But I want to process only if US and IND files are present 

Community Manager

Re: check for specific and latest file, if found then process files one by one

That is fine. That does not have to be part of the ordering. Use a tFileList with the settings I suggested for the ordering and you can use a couple of filemasks to filter IND and US in if you like.....I should have suggested that in the first comment :-)

Cloud Free Trial

Try Talend Cloud free for 30 days.

Tutorial

Introduction to Talend Open Studio for Data Integration.

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Definitive Guide to Data Quality

Create systems and workflow to manage clean data ingestion and data transformation.