tFTPGet - Super Slow

Nine Stars

tFTPGet - Super Slow

1. Is there a component faster than tFTPGet?

2. Is there some setting which can make the tFTPGet transfer speed faster?

3. Is there some config setting which will improve the transfer speed?

 

I am using TOS Version: 7.1.1

 

We have daily files which are 34GB compressed to FTP from an AWS external server which I am told is not limited.

Using the tFTPGet component I can get the file, the issue is the transfer speed is less than 1MB per second.

When I transfer the files via CyberDuck I am seeing speeds of 7.8MB per second, so it takes around an hour per file.

Highlighted
Forteen Stars

Re: tFTPGet - Super Slow

hi,

 

there are no special settings for this component 

you could test with command line scp (winscp in case of windows), if it will be also faster - then use tSystem for call command line utility

 

 

-----------
Highlighted
Nine Stars

Re: tFTPGet - Super Slow

With winSCP I am getting between 1.5MB and 2MB per second.

Highlighted
Forteen Stars

Re: tFTPGet - Super Slow

as I understand CyberDuck support multithread for downloading (multiple-connections-for-file-transfers )

winscp - not

so you can check by searching command line clients like - https://aria2.github.io/

or test command line version of CyberDuck - https://duck.sh/

 

 

-----------
Highlighted
Nine Stars

Re: tFTPGet - Super Slow

I used tLibraryLoad to load edtftpj-2.0.4.jar then in tJava:

 

((com.enterprisedt.net.ftp.FTPClient)globalMap.get("conn_tFTPConnection_1")).setNetworkBufferSize(65536);
((com.enterprisedt.net.ftp.FTPClient)globalMap.get("conn_tFTPConnection_1")).setTransferBufferSize(65536);

 

Error:

Detail Message: The method setNetworkBufferSize(int) is undefined for the type FTPClient

Highlighted
Forteen Stars

Re: tFTPGet - Super Slow

why you manually load library, if it already used by component?

 

tFTPLib.PNG

 

and why you are use libray from 2009?!!

Version 2.0.4
(13 February 2009)
-----------
Highlighted
Nine Stars

Re: tFTPGet - Super Slow

If I don't I am getting this error:

Detail Message: com.enterprisedt cannot be resolved to a type

 

I downloaded edtftpj-2.5.0 to use with tLLibraryLoad, now I get:

java.lang.ClassCastException: com.jcraft.jsch.ChannelSftp cannot be cast to com.enterprisedt.net.ftp.FTPClient

Highlighted
Forteen Stars

Re: tFTPGet - Super Slow

my question was - why you are use outdated library?

 

in any case starting from begin of the thread, I not sure increasing only this parameter resolve your problem

 

Duck work fast because run several threads in parallel, in description of library I not found nothing about this 

-----------
Highlighted
Nine Stars

Re: tFTPGet - Super Slow

The duck.sh website is blocked by our firewall.

Is this what you are suggesting?
https://blog.cyberduck.io/2015/06/24/scripting-cloud-storage-copy-using-the-command-line-interface-c...

Highlighted
Forteen Stars

Re: tFTPGet - Super Slow

yes, but just as variant

 

search google - multithread FTP command line client and test what better for you

-----------

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Best Practices for Using Context Variables with Talend – Part 2

Part 2 of a series on Context Variables

Blog

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog