How to get latest files

Five Stars

How to get latest files

How to get the latest file. when we run history. Can anyone please help to sort it out.


Accepted Solutions
Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files

Just have to change the tJava code to use BigInteger instea of Int:

BigInteger dateFile = new BigInteger(((String)globalMap.get("tFileList_2_CURRENT_FILE")).replaceAll("^.*_", "").replace(".psv", ""));

if ((BigInteger)globalMap.get("maxDate") == null || dateFile.compareTo((BigInteger)globalMap.get("maxDate")) == 1) {
	globalMap.put("maxDate", dateFile);
	globalMap.put("lastFile", (String)globalMap.get("tFileList_2_CURRENT_FILE"));
}

 


TRF

All Replies
Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files

Have a look to tFileList component options to sort files based on chronological order with older first.

TRF
Five Stars

Re: How to get latest files

There is only modified in tfilelist. 

In my scenario I'm downloading all the history files in job server, but based on filedate it should process only latest date file.

Eg: 1.Customersdata_20180126.psv

       2.Customersdata_20180201.psv

       3.Customersdata_20171231.psv

We have to process only  2.Customersdata_20180201.psv file which is latest

 

Thanks,

Venkat

 

Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files

Same answer based on filename.
Memorize the 1rst name using a global variable then reuse the variable in the next subjob to get the file.

TRF
Five Stars

Re: How to get latest files

Can you please explain in detail, because file names may vary.

Forteen Stars

Re: How to get latest files

first put in the memory that the pattern of the file. and call the same pattern in to tFilelist.

 

1) pattern would be like this globalMap.put("pattern", "Customersdata_"+ TalendDate.getDate("yyyyMMdd") + pattern_suffixe);

2) use the same paatern to get see the below screen shot.Untitled.png

 

 

 

--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files

Hi,

Stricly based on your filenames, here is a complete solution to get the latest file:

Capture.PNG

tFileList is configured with the filemask "*.psv" and here is the content of the tJava component:

 

 

// get the date contained into the current filename
int dateFile = Integer.parseInt(((String)globalMap.get("tFileList_2_CURRENT_FILE")).replaceAll("^.*_", "").replace(".psv", ""));

// if there is no "maxDate" known or the current date is > to "maxDate" then the current date becomes the "maxDate"
if ((Integer)globalMap.get("maxDate") == null || dateFile > (Integer)globalMap.get("maxDate")) {
globalMap.put("maxDate", dateFile);
globalMap.put("lastFile", (String)globalMap.get("tFileList_2_CURRENT_FILE"));
}

Now you can get the filename using the global variable called "lastFile" using the following syntax (for example to populate the Filename field of a tFileInputDelimited or tFileInputFullRow component):

(String)globalMap.get("lastFile")

This example is based on your input and works fine.

You may have to adapt it to your case.

Don't forget to mark your case as solved as soon as it is ok for you (Kudo also accepted).


TRF
Five Stars

Re: How to get latest files

Hi,

I'm very thankful for your detailed explanation.

My filename is coming like this

 

Good_Customersdata_20171231_20171231092635.psv

In this file 20171231092635 is date with timestamp I can to check based on this.

Please help me to sort it 

Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files

Just have to change the tJava code to use BigInteger instea of Int:

BigInteger dateFile = new BigInteger(((String)globalMap.get("tFileList_2_CURRENT_FILE")).replaceAll("^.*_", "").replace(".psv", ""));

if ((BigInteger)globalMap.get("maxDate") == null || dateFile.compareTo((BigInteger)globalMap.get("maxDate")) == 1) {
	globalMap.put("maxDate", dateFile);
	globalMap.put("lastFile", (String)globalMap.get("tFileList_2_CURRENT_FILE"));
}

 


TRF
Five Stars

Re: How to get latest files

It is showing BigInt cannot be resolved.

Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files

Add the following in the tJava component Advanced settings:

import java.math.*;

TRF
Forteen Stars

Re: How to get latest files

did you try my solution?

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Five Stars

Re: How to get latest files

Thanks for your support

Actually I was confused, I tried but didn't understand 

Forteen Stars

Re: How to get latest files

1) in tjava put below code.

String pattern_suffixe = "*.csv";

globalMap.put("yyyyMMdd", TalendDate.getDate("yyyy_MM_dd"));

globalMap.put("pattern", "Customersdata_"+ TalendDate.getDate("yyyyMMdd") + pattern_suffixe);

2) in tFilelist use the below parameter under the filemask.same which i have provided in screenshot in the first reply.

((String)globalMap.get("pattern"))

 

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Forteen Stars

Re: How to get latest files

is my solution usefull?

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Fifteen Stars TRF
Fifteen Stars

Re: How to get latest files

Finally you will be able to get the file with using the variable "lastFile" as a filename:

Capture.PNG

Now you got a complete solution for your case.

Don't forget to mark it as solved (Kudos also accepted).


TRF
Five Stars

Re: How to get latest files

Hi ,

 

Thanks for your support, I will accept your solution.

I found one simple way to get the latest file.

In tfilelist  we have by filename instead of default and order by des.

Then we will get latest file.

 

Thanks

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Definitive Guide to Data Integration

Practical steps to developing your data integration strategy.

Download