One Star

Retrieving Latest Files From FTP (Java)

I have an FTP account from which I must retrieve the most recent version for four different files. Every day our data source posts four different files with a datestamp appended to the end of the filename... Ex:
idns.txt.11310
lim.txt.11310
sbo.txt.11310
epf.txt.11310
NOTE: The datestamp is mmddyy where mm is a single or double digit, dd is the day, yy is the two-digit year.
If my process does not run for 5 days, I'll end up with 20 files in the directory.
I'm assuming I'll need to use a tFTPFileList and potentially tFileProperties to iterate over all files, but how do I compare datestamps to obtain the 'most recent' version of each of the four file types?
12 REPLIES
One Star

Re: Retrieving Latest Files From FTP (Java)

I neglected to mention that I also need to truncate the ".mmddyy" extension upon retrieval to work on the ".txt" files.
Community Manager

Re: Retrieving Latest Files From FTP (Java)

Hello Andybrown
Here I show you a demo job, feel free to post your question if you have any trouble.
code on tJavaRow_1:
//Code generated according to input schema and output schema
if(input_row.date.length()==5){ //like: 11301
output_row.basename = input_row.basename;
output_row.date = TalendDate.parseDate("Mddyy",input_row.date);
}
else{ //the length of date is 6, like 111209
output_row.basename = input_row.basename;
output_row.date = TalendDate.parseDate("MMddyy",input_row.date);
}

the destination filename on tFIleCopy_1:
((String)globalMap.get("tFileList_1_CURRENT_FILE")).substring(0,((String)globalMap.get("tFileList_1_CURRENT_FILE")).lastIndexOf("."))

About usage of tFlowToIterate component, see 5827
Best regards

shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Retrieving Latest Files From FTP (Java)

Hi Shong,
I greatly appreciate your time in explaining this to me. It's now perfectly understandable. However, I'm getting the following exception on tFTPFileProperties_1...
======
Starting job Get_Feed_Files at 06:06 14/01/2010.
connecting to socket on port 3334
connected
Exception in component tFTPFileProperties_1
com.enterprisedt.net.ftp.FTPException: ./lim.txt.11310: file too large for SIZE when TYPE is ASCII.
at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:999)
at com.enterprisedt.net.ftp.FTPClient.size(FTPClient.java:1953)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPFileList_1Process(Get_Feed_Files.java:1792)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPConnection_1Process(Get_Feed_Files.java:482)
at mfr.get_feed_files_0_1.Get_Feed_Files.runJobInTOS(Get_Feed_Files.java:2686)
at mfr.get_feed_files_0_1.Get_Feed_Files.main(Get_Feed_Files.java:2522)
578 milliseconds
disconnected
Job Get_Feed_Files ended at 06:06 14/01/2010.
======
The actual size of the file is just 41,045 bytes (yes, just 41.0 KB). This is happening on both active and passive modes. Any ideas?
Thanks,
-Andy
One Star

Re: Retrieving Latest Files From FTP (Java)

Another note... after changing all connection modes to "active" from the initial connection on down, I'm getting this exception:
=====
Starting job Get_Feed_Files at 06:14 14/01/2010.
connecting to socket on port 3742
connected
Exception in component tFTPFileList_1
java.io.IOException: Unexpected null reply received
at com.enterprisedt.net.ftp.FTPControlSocket.readReply(FTPControlSocket.java:855)
at com.enterprisedt.net.ftp.FTPClient.dir(FTPClient.java:3089)
at mfr.get_feed_files_0_1.Get_Feed_Files.tFTPFileList_1Process(Get_Feed_Files.java:1642)
at mfrget_feed_files_0_1.Get_Feed_Files.tFTPConnection_1Process(Get_Feed_Files.java:482)
at mfr.get_feed_files_0_1.Get_Feed_Files.runJobInTOS(Get_Feed_Files.java:2686)
at mfr.get_feed_files_0_1.Get_Feed_Files.main(Get_Feed_Files.java:2522)
1360 milliseconds
disconnected
Job Get_Feed_Files ended at 06:14 14/01/2010.
=======
Four Stars

Re: Retrieving Latest Files From FTP (Java)

You have to change the file type in the components setting to Binary from Ascii...
One Star

Re: Retrieving Latest Files From FTP (Java)

You have to change the file type in the components setting to Binary from Ascii...

Thanks for the response...
That was my first guess based on the exception, however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).
Community Manager

Re: Retrieving Latest Files From FTP (Java)

Hello
however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).

The option is on tFTPGet component, so now only use tFTPGet to test if you can get the file to local machine.
Best regards
shong
----------------------------------------------------------
Talend | Data Agility for Modern Business
One Star

Re: Retrieving Latest Files From FTP (Java)

Hello
however there are no options for "Binary | Ascii" in any of the Java FTP controls. There is only 'mode' (passive vs. active).

The option is on tFTPGet component, so now only use tFTPGet to test if you can get the file to local machine.
Best regards
shong

Unfortunately, I am having major issues with the tFTPGet component in retrieving files from my FTP server. I have a ticket open and a WebEx with Tim Drees next week about this.
In the meantime, is there any other way to set the mode to binary in the tFTPFileProperties component, since this is the component raising the exception?
Four Stars

Re: Retrieving Latest Files From FTP (Java)

A quick work around would be to use tFtpGet to download the files, do the file check locally (using tFileList) and then send back to another ftpGet which files you want to download.
One Star

Re: Retrieving Latest Files From FTP (Java)

What I ended up doing is to bring the files down to a temp directory using a tFTPFileList and a tSystem (using a curl command for each file), then iterating over the local files, moving them to my inbound directory. I'll delete the source files both locally and on the FTP server so I only have to process a handful each time. Until my FTP issue is identified, this shall be it.
Thanks for the assist.
Four Stars

Re: Retrieving Latest Files From FTP (Java)

Would love to check your job out if you're willing to share. Sounds like a good work around...
One Star

Re: Retrieving Latest Files From FTP (Java)

hey,
please i have problem
my team is working at BI prject with talend to retrieve data from datasource using ftp servers to another place, please if anyone can help me and tell me how this processus work ??