How to get latest files from folder based on last run

Five Stars

How to get latest files from folder based on last run

How to get incremental files from same folder …in talend

I have to get files based on last job start date after which are files coming those files i need process..how i can achive this.pls help me..Thank you


Accepted Solutions
Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files from folder based on last run

The simplest (the better?) is to move (using tfilecopy) the files to a special folder (archive for example) as soon they have been processed. Using this pattern, you don't have to worry about how many files you have processed last day. Just iterate over the filelist and that's all.

TRF
Seven Stars

Re: How to get latest files from folder based on last run

here is how to do that, we had same requirement on getting latest file from FTP,

 

basically we compare the files creation date & compare it with previous files creation date & that's how we basically sort the file & get latest file,

 

globalMap.get("sortedmtime") == null ? row1.mtime : (row1.mtime >= (long)globalMap.get("sortedmtime"))?row1.mtime:globalMap.get("sortedmtime")

image.png

 

There are so many other solution out there like using tbuffer to store all files & then use tSort to sort data


All Replies
Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files from folder based on last run

Store the last filename processed in a file (and reuse this information for next run) or rename the files after they have been processed.

TRF
Five Stars

Re: How to get latest files from folder based on last run

Hi TRF, Thanks for your response....My requirement is incremental files are coming to client server from that we need process the those files, which are newly coming from last run..

 

Ex: Yesterday 10 files came ---> we processed--

 

today 5 files came ---->total 15(yesterday 10+today 5)--from this i want to get latest files and need to process..

 

 

How we can this?

 

Please share the complete Talend Design ?

Thank you so much 

Seven Stars

Re: How to get latest files from folder based on last run

Can you also send your file name format?? I am also facing same issue so,
I came up with a solution where i am getting all in tfilelist and iterate those file from tJava (compare those files and get next file , b4 that i will store last run job file name)
Can anyone give me better solution.. Thanks in Advance!!
Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files from folder based on last run

The simplest (the better?) is to move (using tfilecopy) the files to a special folder (archive for example) as soon they have been processed. Using this pattern, you don't have to worry about how many files you have processed last day. Just iterate over the filelist and that's all.

TRF
Highlighted
Five Stars

Re: How to get latest files from folder based on last run

Hi Sir, Thank you for your response..but my client saying without achieve the files we need to process latest files based on last run 

Seven Stars

Re: How to get latest files from folder based on last run

here is how to do that, we had same requirement on getting latest file from FTP,

 

basically we compare the files creation date & compare it with previous files creation date & that's how we basically sort the file & get latest file,

 

globalMap.get("sortedmtime") == null ? row1.mtime : (row1.mtime >= (long)globalMap.get("sortedmtime"))?row1.mtime:globalMap.get("sortedmtime")

image.png

 

There are so many other solution out there like using tbuffer to store all files & then use tSort to sort data

Five Stars

Re: How to get latest files from folder based on last run

Thanks Nmodi for your valuable solution..is it working for TOS?

 

I will try and update you.

 

Thanks

Four Stars

Re: How to get latest files from folder based on last run

Hi All,

 

we can do with combination of tFileList and tFileExist components.

  1. Let us Assume A1 is the Source.
  2. Create A2 and A3 two other folders.
  3. Iterate A1 folders with tFileList and use tFileExists -A2 directory path and global variable "tFileList_1_CURRENT_FILE to check the files in A2.
  4. After TfileExistsuse condition tFileExist_2_EXISTS then copy the file using tFileCopy to Folder A3.
  5. At end of run make sync between A1 and A2.
  6. After processing new files in A3 archive/delete them 
  7. Like this every run new files will be stored in A3.

 

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Tutorial

Introduction to Talend Open Studio for Data Integration.

Watch

Downloads and Trials

Test drive Talend's enterprise products.

Downloads

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download