Get Latest Data File?

One Star

Get Latest Data File?

Is there a routine that can be written to automatically get the latest file or the file for the previous date? Just as 'getDate' allows you to retrieve the file for today automatically.

Example: Today is 20070724 and i want to pull the latest file available. Below are example files. The file I would want to return would be the ABC20070710 which is the latest file.
ABC20070623
ABC20070626
ABC20070630
ABC20070710
Employee Xin
Employee

Re: Get Latest Data File?

yes, in job design->code->routine
I?ll give you a simple example
//template routine Java
package routines;
public class MyRoutine {
public static String getLastestFile(String directory) {
// read Directory and get the last File
return "ABC20070710";
}
}
Any component can use this routine, for instance, tMsBox, in the message testbox, type MyRoutine. getLastestFile ("c:/mydir/"),after run the job you?ll see the how routine is used.
This is a simple example of how to use routine, if you want to implement your case, some programming is needed
One Star

Re: Get Latest Data File?

Hi xliu,
wouldn't it be a nice feautere if tFileList exports a main-row (additional to the "iterate"), with file information :-)
So this problem wouldn't need any special programming (tFileList -> tAggregateRow).
Bye
Volker
One Star

Re: Get Latest Data File?

Hi xliu,
Is this routine applicable to Perl? I have written these jobs in Perl. Also, in the routine you have look for a specific date "ABC20070710". I want to automate the job so I find the latest file each day. In the example of files listed the "ABC20070710" was the latest file but if today a file "ABC20070717" shows up that would be the file I would want returned.
One Star

Re: Get Latest Data File?

Hello Rodney,
this solution is working only for java-generated code. Xliu please correct me, if I am wrong.
If you generat perl code, you have to code in perl. I have actual no talend software running, so I can actual give you only some simple advice:
Write a new perl-routine which returns the correct filename (like xliu describes for java)
Input: $path, Output: $fileWithPathname
I don't know how firm you are with perl.
You can get a list of the files with glob or the <>-operator (<filemask>).
Process this list and extract the relevant filename.
Possibly the following module can help you:
http://search.cpan.org/~yves/Date-Simple-3.02/lib/Date/Simple.pm
Bye
Volker
Employee Xin
Employee

Re: Get Latest Data File?

Hi,Volker Brehm
Component tFileList can't be linked to tAggregateRow with Iterate,Because tFileList does not have schema, while tAggregateRow does, so the job can not run to acheive this.
Cheers
Highlighted
One Star

Re: Get Latest Data File?

Hi xliu,
the link between tFileList and tAggregateRow (or any other component) was just a idea. I thought about opening a feauture request in the Bugtracker.
Bye
Volker
One Star

Re: Get Latest Data File?

Hi xliu,
Is there a routine applicable to Perl? I have written these jobs in Perl. Also, in the routine you have look for a specific date "ABC20070710". I want to automate the job so I find the latest file each day. In the example of files listed the "ABC20070710" was the latest file but if today a file "ABC20070717" shows up that would be the file I would want returned.
Employee

Re: Get Latest Data File?

Hi,
This topic will probably help you 247
Hope this helps

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now