download latest files from ftp server

Highlighted
Six Stars

download latest files from ftp server

I downloaded files for last 3 months from ftp server. These are the past files. I need to download the future files every day from the ftp server. The previous files file_2018_07_01 to file_2018_09_09 are downloaded but if I have to download file_2018_09_10 without looking for the file name, how do I do it? There will be multiple files like file_2018_09_10 everyday that have to be downloaded. How to compare the list of files between PC and FTP server, fetch files that are not on PC?

Can anyone suggest the right components for it?

Thanks.

Ten Stars

Re: download latest files from ftp server

tFileList and tFTPFileList,
You need to iterate over and add each value (filename) to a file or memory.
You could compare both list with a tMap and join,
Non matching files (those who are not there yet) iterate tFTPGet.

Alternative :
1 - Maybee its more easier to download and filter based on a date and is the most common approach for 'syncing' folders.
Or Alternative:
2 - You move the dowloaded files in a'AllreadyDownload" folder on you FTP server so you have a clean folder where new files are placed.

Keep it simple.
Forteen Stars

Re: download latest files from ftp server

@tychobrahe,you want to download the correspond date file on every day basis? if yes you can do with below approcah

in filemask of tFTPGet.

 

"*" + TalendDate.getDate("yyyy_MM_dd")+"*"

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Six Stars

Re: download latest files from ftp server

@manodwhb thank you! the issue could be that there are many data files that get uploaded on ftp server everyday and a lot of them aren't needed to be downloaded. So, is there another kind of filter that we can use? Something other than filename format as the format could be changed every year.

Six Stars

Re: download latest files from ftp server

@Dijke, thank you very much. 2nd alternative is ruled out as we can't do that on FTP server.

Regarding the main approach you gave using tFTPFileList, tFileList and then using tFTPGet, would that be possible to download the ones that are not matching between tFileList & tFTPFileList? Would it work even if the file names are changed sometime by the FTP admin? There is a possibility that they may change.

1st alternative with syncing dates, could you be more specific? I'm pretty new to Talend and could use some more details.

Six Stars

Re: download latest files from ftp server

Spoiler
 

 @manodwhb the job works but no files fetched. 

Forteen Stars

Re: download latest files from ftp server

@tychobrahe,the pattern that you provided has not satisfy to fetch the file. please check the pattern.

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Forteen Stars

Re: download latest files from ftp server

@tychobrahe,still do you have issue?

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Six Stars

Re: download latest files from ftp server

@tychobrah 

If you are not sure about the filename , date format then the only way would be to compare the file content between the files present in the local and ftp server . you can use tfilecompare which would tell you whether there is a difference between local and ftp server files and then you can download files which only have a difference.

 

Thanks ,

Praveen M.

Six Stars

Re: download latest files from ftp server

@tychobrah 

Alternatives,

 

1.You can use 'exclude file mask' option on advanced settings of tfilelist , which does not include the files falling under such mask , where you can iterate over the filenames that are already present in the local and put the entire filename as mask on the the excludeFileMAsk option and then download the ones which come through all the exclude file masks.

2.Use tfileExist after tftpList by which you can check whether the ftp file is present in the local and if not present then only pass it to ftpGet using trigger-RunIf link giving the if clause.

 

Thanks,

Praveen M.

 

 

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now