Five Stars

How to get latest files

How to get the latest file. when we run history. Can anyone please help to sort it out.

1 ACCEPTED SOLUTION

Accepted Solutions
Thirteen Stars TRF
Thirteen Stars

Re: How to get latest files

Just have to change the tJava code to use BigInteger instea of Int:

BigInteger dateFile = new BigInteger(((String)globalMap.get("tFileList_2_CURRENT_FILE")).replaceAll("^.*_", "").replace(".psv", ""));

if ((BigInteger)globalMap.get("maxDate") == null || dateFile.compareTo((BigInteger)globalMap.get("maxDate")) == 1) {
	globalMap.put("maxDate", dateFile);
	globalMap.put("lastFile", (String)globalMap.get("tFileList_2_CURRENT_FILE"));
}

 


TRF
16 REPLIES
Thirteen Stars TRF
Thirteen Stars

Re: How to get latest files

Have a look to tFileList component options to sort files based on chronological order with older first.

TRF
Five Stars

Re: How to get latest files

There is only modified in tfilelist. 

In my scenario I'm downloading all the history files in job server, but based on filedate it should process only latest date file.

Eg: 1.Customersdata_20180126.psv

       2.Customersdata_20180201.psv

       3.Customersdata_20171231.psv

We have to process only  2.Customersdata_20180201.psv file which is latest

 

Thanks,

Venkat

 

Thirteen Stars TRF
Thirteen Stars

Re: How to get latest files

Same answer based on filename.
Memorize the 1rst name using a global variable then reuse the variable in the next subjob to get the file.

TRF
Five Stars

Re: How to get latest files

Can you please explain in detail, because file names may vary.

Ten Stars

Re: How to get latest files

first put in the memory that the pattern of the file. and call the same pattern in to tFilelist.

 

1) pattern would be like this globalMap.put("pattern", "Customersdata_"+ TalendDate.getDate("yyyyMMdd") + pattern_suffixe);

2) use the same paatern to get see the below screen shot.Untitled.png

 

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Manohar B
Thirteen Stars TRF
Thirteen Stars

Re: How to get latest files

Hi,

Stricly based on your filenames, here is a complete solution to get the latest file:

Capture.PNG

tFileList is configured with the filemask "*.psv" and here is the content of the tJava component:

 

 

// get the date contained into the current filename
int dateFile = Integer.parseInt(((String)globalMap.get("tFileList_2_CURRENT_FILE")).replaceAll("^.*_", "").replace(".psv", ""));

// if there is no "maxDate" known or the current date is > to "maxDate" then the current date becomes the "maxDate"
if ((Integer)globalMap.get("maxDate") == null || dateFile > (Integer)globalMap.get("maxDate")) {
globalMap.put("maxDate", dateFile);
globalMap.put("lastFile", (String)globalMap.get("tFileList_2_CURRENT_FILE"));
}

Now you can get the filename using the global variable called "lastFile" using the following syntax (for example to populate the Filename field of a tFileInputDelimited or tFileInputFullRow component):

(String)globalMap.get("lastFile")

This example is based on your input and works fine.

You may have to adapt it to your case.

Don't forget to mark your case as solved as soon as it is ok for you (Kudo also accepted).


TRF
Five Stars

Re: How to get latest files

Hi,

I'm very thankful for your detailed explanation.

My filename is coming like this

 

Good_Customersdata_20171231_20171231092635.psv

In this file 20171231092635 is date with timestamp I can to check based on this.

Please help me to sort it 

Thirteen Stars TRF
Thirteen Stars

Re: How to get latest files

Just have to change the tJava code to use BigInteger instea of Int:

BigInteger dateFile = new BigInteger(((String)globalMap.get("tFileList_2_CURRENT_FILE")).replaceAll("^.*_", "").replace(".psv", ""));

if ((BigInteger)globalMap.get("maxDate") == null || dateFile.compareTo((BigInteger)globalMap.get("maxDate")) == 1) {
	globalMap.put("maxDate", dateFile);
	globalMap.put("lastFile", (String)globalMap.get("tFileList_2_CURRENT_FILE"));
}

 


TRF
Five Stars

Re: How to get latest files

It is showing BigInt cannot be resolved.

Thirteen Stars TRF
Thirteen Stars

Re: How to get latest files

Add the following in the tJava component Advanced settings:

import java.math.*;

TRF
Ten Stars

Re: How to get latest files

did you try my solution?

Manohar B
Five Stars

Re: How to get latest files

Thanks for your support

Actually I was confused, I tried but didn't understand 

Ten Stars

Re: How to get latest files

1) in tjava put below code.

String pattern_suffixe = "*.csv";

globalMap.put("yyyyMMdd", TalendDate.getDate("yyyy_MM_dd"));

globalMap.put("pattern", "Customersdata_"+ TalendDate.getDate("yyyyMMdd") + pattern_suffixe);

2) in tFilelist use the below parameter under the filemask.same which i have provided in screenshot in the first reply.

((String)globalMap.get("pattern"))

 

Manohar B
Ten Stars

Re: How to get latest files

is my solution usefull?

Manohar B
Thirteen Stars TRF
Thirteen Stars

Re: How to get latest files

Finally you will be able to get the file with using the variable "lastFile" as a filename:

Capture.PNG

Now you got a complete solution for your case.

Don't forget to mark it as solved (Kudos also accepted).


TRF
Five Stars

Re: How to get latest files

Hi ,

 

Thanks for your support, I will accept your solution.

I found one simple way to get the latest file.

In tfilelist  we have by filename instead of default and order by des.

Then we will get latest file.

 

Thanks