Dear all, I download files from a ftp server. See my Job design attached. What I want to do in detail: On FTP following structure is given: dir1 -file1 -file2 dir2 -file4 -flie5 I want to copy the whole directory structure. file 1 and file 2 should be deposit in dir1 on my local server etc. That is the reason why I use two FTP FileLists. FtpFileList_1 reads dir1, dir2 etc. and FtpFileList_2 reads the files within the directories. So I'm able to re-use the component parameter to determine the destination directory (see GetZipsFromIn). Second, I want to make sure the upload of the files is done - that's why I use FTPFileProperty (mtime) and filter for it. Furthermore I use FtpRename (RenameToPROGRESS) to rename the file before I download it (regard iteration order). This makes sure that a second job (I use cronjob) won't download a file that is currently downloaded by the first job (file mask in FtpFileList_2). Anyway, everything works perfect. The job downloaded up to 90 files. But whenever the download took more than 5 minutes, I got error messages (see below). I tested it out with one huge file and lots of small/normal files. So too many connections won't be a problem. Strangely the error occurs after the file(s) has been downloaded even when it took more than 5 minutes! But it leads to an aboration of the job! Parallel I tested it out with FileZilla (same server, same user). There I didn't have any trouble. So FTP server don't have any restrictions. Also I modfied the job several times - I got rid of the flowtoiterate component or modified iterations to flows. But every time, I got the same error. I cannot use the ftp connection component, because each component needs a "direct" connection. Any ideas? It might be a problem to use the component parameter too often.. Error message (1 big file): Exception in component tFTPFileProperties_1 com.enterprisedt.net.ftp.FTPConnectionClosedException: Idle timeout (300 seconds): closing control connection at com.enterprisedt.net.ftp.FTPControlSocket.validateReplyCode(FTPControlSocket.java:1217) at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1167) at com.enterprisedt.net.ftp.FTPClient.quit(FTPClient.java:3949) at submissionchecker.tesdsg4_0_1.tesdsg4.tFTPFileList_1Process(tesdsg4.java:1806) at submissionchecker.tesdsg4_0_1.tesdsg4.runJobInTOS(tesdsg4.java:2503) at submissionchecker.tesdsg4_0_1.tesdsg4.main(tesdsg4.java:2368) Error message (multiple files with various file size; 0,5 MB - 40 MB): Exception in component tFTPFileList_1 com.enterprisedt.net.ftp.FTPConnectionClosedException: No transfer timeout (300 seconds): closing control connection at com.enterprisedt.net.ftp.FTPControlSocket.validateReplyCode(FTPControlSocket.java:1217) at com.enterprisedt.net.ftp.FTPControlSocket.validateReply(FTPControlSocket.java:1167) at com.enterprisedt.net.ftp.FTPClient.quit(FTPClient.java:3949) at submissionchecker.executetranspect_0_3.ExecuteTranspect.tFTPFileList_2Process(ExecuteTranspect.java:4714) at submissionchecker.executetranspect_0_3.ExecuteTranspect.tFixedFlowInput_1Process(ExecuteTranspect.java:3747) at submissionchecker.executetranspect_0_3.ExecuteTranspect.runJobInTOS(ExecuteTranspect.java:6090) at submissionchecker.executetranspect_0_3.ExecuteTranspect.runJob(ExecuteTranspect.java:5936) at submissionchecker.master_0_1.Master.tRunJob_1Process(Master.java:726) at submissionchecker.master_0_1.Master.tRunJob_3Process(Master.java:572) at submissionchecker.master_0_1.Master.runJobInTOS(Master.java:995) at submissionchecker.master_0_1.Master.main(Master.java:839)
I found out the reason why the connection was terminated. The reason is the FTP server (TransferTimout, TransferIdle). I increased the time to more than five minutes and it worked. For me this inidicates that a not-in-use ftp connection is all the time open (it might be the ftp file lists components). This would not be a problem for the download itself, but for the job (because I cannot capture the error message even when I use the subjob error or component error link).
Hi _OZ_ thats because I have recently published a tFTPClose component because the tFTPConnection component or an other FTP components does not close the connection. Only the death of the job closes the connection.
Dear jlolling, thanks for your reply. I downloaded the ftpclose component too. But I only can use it, when i pick up a ftpconnection component. But using a ftpconnection component doesn't make sense, because I have to open a connection for each component (ftpfilelist, ftpfileproperty etc.) separately. Or does it mean ftpclose closes all open connection even when it is a connection opened with other ftp components?
For your luck, the tFTPget/put/list connections closes it internal connections for it self. Take a look at the source code and scroll to the end part and you will find the close/disconnected call. The tFTPClose closes ONLY the connection which are established from a tFTPConnection component, anything else does not make sense. BUT: If you start transfers within a loop or iteration take care you do not establish to much connections: tFileList --(iteration)--> tFTPPut (creates it own connection for every file!) this will cause probably a kind of problem you have. You can adapt this example also for the get operation.
Make sense that each component closes the ftp connection for itself. I guess the problem is the FTPFileList. It will keep the connection alive until all list objects (files/directories) are processed respectively until the subjob is done (which takes in my case a while). Therefore the only solution is to increase the server params... I can live with that solution, but honestly it would be better to capture/to get rid of the error message or the redesign the job (but honestly I have no idea how to design it without a FtpFileList)... Thanks again for ideas/help. I appreciate it.