[resolved] FTP timeout

Highlighted
One Star

[resolved] FTP timeout

Dear all,
I download files from a ftp server. See my Job design attached.
What I want to do in detail:
On FTP following structure is given:
dir1
-file1
-file2
dir2
-file4
-flie5
I want to copy the whole directory structure. file 1 and file 2 should be deposit in dir1 on my local server etc. That is the reason why I use two FTP FileLists. FtpFileList_1 reads dir1, dir2 etc. and FtpFileList_2 reads the files within the directories. So I'm able to re-use the component parameter to determine the destination directory (see GetZipsFromIn).
Second, I want to make sure the upload of the files is done - that's why I use FTPFileProperty (mtime) and filter for it.
Furthermore I use FtpRename (RenameToPROGRESS) to rename the file before I download it (regard iteration order). This makes sure that a second job (I use cronjob) won't download a file that is currently downloaded by the first job (file mask in FtpFileList_2). Anyway, everything works perfect.
The job downloaded up to 90 files. But whenever the download took more than 5 minutes, I got error messages (see below). I tested it out with one huge file and lots of small/normal files. So too many connections won't be a problem.
Strangely the error occurs after the file(s) has been downloaded even when it took more than 5 minutes! But it leads to an aboration of the job!
Parallel I tested it out with FileZilla (same server, same user). There I didn't have any trouble. So FTP server don't have any restrictions.
Also I modfied the job several times - I got rid of the flowtoiterate component or modified iterations to flows. But every time, I got the same error. I cannot use the ftp connection component, because each component needs a "direct" connection.
Any ideas? It might be a problem to use the component parameter too often..
Error message (1 big file):
Exception in component tFTPFileProperties_1
com.enterprisedt.net.ftp.FTPConnectionClosedException: Idle timeout (300 seconds): closing control connection
at com.enterprisedt.net.ftp.FTPControlSocket.validateReplyCode(FTPControlSocket.java:1217)
at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1167)
at com.enterprisedt.net.ftp.FTPClient.quit(FTPClient.java:3949)
at submissionchecker.tesdsg4_0_1.tesdsg4.tFTPFileList_1Process(tesdsg4.java:1806)
at submissionchecker.tesdsg4_0_1.tesdsg4.runJobInTOS(tesdsg4.java:2503)
at submissionchecker.tesdsg4_0_1.tesdsg4.main(tesdsg4.java:2368)
Error message (multiple files with various file size; 0,5 MB - 40 MB):
Exception in component tFTPFileList_1
com.enterprisedt.net.ftp.FTPConnectionClosedException: No transfer timeout (300 seconds): closing control connection
at com.enterprisedt.net.ftp.FTPControlSocket.validateReplyCode(FTPControlSocket.java:1217)
at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1167)
at com.enterprisedt.net.ftp.FTPClient.quit(FTPClient.java:3949)
at submissionchecker.executetranspect_0_3.ExecuteTranspect.tFTPFileList_2Process(ExecuteTranspect.java:4714)
at submissionchecker.executetranspect_0_3.ExecuteTranspect.tFixedFlowInput_1Process(ExecuteTranspect.java:3747)
at submissionchecker.executetranspect_0_3.ExecuteTranspect.runJobInTOS(ExecuteTranspect.java:6090)
at submissionchecker.executetranspect_0_3.ExecuteTranspect.runJob(ExecuteTranspect.java:5936)
at submissionchecker.master_0_1.Master.tRunJob_1Process(Master.java:726)
at submissionchecker.master_0_1.Master.tRunJob_3Process(Master.java:572)
at submissionchecker.master_0_1.Master.runJobInTOS(Master.java:995)
at submissionchecker.master_0_1.Master.main(Master.java:839)

Accepted Solutions
Moderator

Re: [resolved] FTP timeout

Hi twincam16,
So far, there is no tFTPClose component in talend. You can vote for this feature jira issue https://jira.talendforge.org/browse/TDI-20134 or have a look at custom component TalendExchange:tFTPClose written by jlolling to see if it is OK with you.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

View solution in original post


All Replies
One Star

Re: [resolved] FTP timeout

I found out the reason why the connection was terminated. The reason is the FTP server (TransferTimout, TransferIdle). I increased the time to more than five minutes and it worked.
For me this inidicates that a not-in-use ftp connection is all the time open (it might be the ftp file lists components). This would not be a problem for the download itself, but for the job (because I cannot capture the error message even when I use the subjob error or component error link).
Seventeen Stars

Re: [resolved] FTP timeout

Hi _OZ_
thats because I have recently published a tFTPClose component because the tFTPConnection component or an other FTP components does not close the connection. Only the death of the job closes the connection.
One Star

Re: [resolved] FTP timeout

Dear jlolling,
thanks for your reply.
I downloaded the ftpclose component too. But I only can use it, when i pick up a ftpconnection component. But using a ftpconnection component doesn't make sense, because I have to open a connection for each component (ftpfilelist, ftpfileproperty etc.) separately.
Or does it mean ftpclose closes all open connection even when it is a connection opened with other ftp components?
Seventeen Stars

Re: [resolved] FTP timeout

For your luck, the tFTPget/put/list connections closes it internal connections for it self. Take a look at the source code and scroll to the end part and you will find the close/disconnected call.
The tFTPClose closes ONLY the connection which are established from a tFTPConnection component, anything else does not make sense.
BUT:
If you start transfers within a loop or iteration take care you do not establish to much connections:
tFileList --(iteration)--> tFTPPut (creates it own connection for every file!)
this will cause probably a kind of problem you have. You can adapt this example also for the get operation.
One Star

Re: [resolved] FTP timeout

Make sense that each component closes the ftp connection for itself.
I guess the problem is the FTPFileList. It will keep the connection alive until all list objects (files/directories) are processed respectively until the subjob is done (which takes in my case a while). Therefore the only solution is to increase the server params... I can live with that solution, but honestly it would be better to capture/to get rid of the error message or the redesign the job (but honestly I have no idea how to design it without a FtpFileList)...
Thanks again for ideas/help. I appreciate it.
One Star

Re: [resolved] FTP timeout

Hi,
You can store the list (from FTPFileList) in a:
- temporary file
- temporary buffer (using the "technical" tHashOutput/tHashInput components)
So your FTP connection will be closed faster
regards
One Star

Re: [resolved] FTP timeout

@svert. Thanks for this hint. A good idea to use the hash components. When I have time, I give it a trial.
One Star

Re: [resolved] FTP timeout

Hi ,
does the tftp get close its connection?
Moderator

Re: [resolved] FTP timeout

Hi twincam16,
So far, there is no tFTPClose component in talend. You can vote for this feature jira issue https://jira.talendforge.org/browse/TDI-20134 or have a look at custom component TalendExchange:tFTPClose written by jlolling to see if it is OK with you.
Best regards
Sabrina
--
Don't forget to give kudos when a reply is helpful and click Accept the solution when you think you're good with it.

View solution in original post

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog