What is the order on which tFTPFileList components decides to iterate the file from FTP server

Six Stars

What is the order on which tFTPFileList components decides to iterate the file from FTP server

Hi,

 

I am looking to get some info on the order on which tFTPFileList component decides to iterate the files from FTP server.

 

I have 3 files in the server with different timing.

File1     :   Apr 17  11:20

File2     :   Apr 17  13:20

File3     :   Apr 17  14:20

 

Which file would be pulled first?

 

When i run my job, files are not getting downloaded based on timing. 

I expect File1 to be pulled first. (First come first serve logic) But that is not happening.

I see "Order By" option in tFileList component but that is not there in tFTPFileList.

 

Any thoughts?

 


Accepted Solutions
Forteen Stars

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

@stamrake,in tFTPFile list we do not have option to sort, since you need to get the files in to local directory,from there you need to list using tFilelist. where in tFilelist you can specify the order by filename or file size or modified date or by default with acesnding or desending.

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.

All Replies
Forteen Stars

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

@stamrake,in tFTPFilelist you do not have "order by"  Option,so first try to get the files in local directory and then list the files using tFileList,where you can specify the order by.

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Six Stars

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

@manodwhb

Thanks Manohar for the idea, 

It's a good workaround however still I am not sure how do i know that which file got pulled to local directory first.

 

If FILE3 gets pulled to local directory first and if I sort by time, it means I would be first processing FILE3 instead of FILE1.

 

Regards,

Sanjay T

Forteen Stars

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

@stamrake,can you sort by modified date on descending order then you may get File1.

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Forteen Stars

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

@stamrake,still do you have an issue?

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Highlighted
Six Stars

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

@manodwhb

Hi, Thanks for the idea.

This trick could give me FILE1 to be processed at first, but there is no guarantee for that, as we do not have sorting option in tFTPFileList.

For my use case i need to be certain of which file is getting processed first. Smiley Happy

 

Regards,

Sanjay

Forteen Stars

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

@stamrake,in tFTPFile list we do not have option to sort, since you need to get the files in to local directory,from there you need to list using tFilelist. where in tFilelist you can specify the order by filename or file size or modified date or by default with acesnding or desending.

Manohar B
Don't forget to give kudos/accept the solution when a replay is helpful.
Six Stars

Re: What is the order on which tFTPFileList components decides to iterate the file from FTP server

@manodwhb

Hi,

 

I agree with your suggestion for the work around.

Guess what, I have requested Talend Enterprise team to have a feature request on the component tFTPFileList to have "Order By" Option.

Feature request detail: (TDI-40435)

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Why Companies Move to the Cloud: 7 Success Stories

Learn how and why companies are moving to the Cloud

Read Now