tFTPGet - Super Slow

Highlighted
Nine Stars

tFTPGet - Super Slow

1. Is there a component faster than tFTPGet?

2. Is there some setting which can make the tFTPGet transfer speed faster?

3. Is there some config setting which will improve the transfer speed?

 

I am using TOS Version: 7.1.1

 

We have daily files which are 34GB compressed to FTP from an AWS external server which I am told is not limited.

Using the tFTPGet component I can get the file, the issue is the transfer speed is less than 1MB per second.

When I transfer the files via CyberDuck I am seeing speeds of 7.8MB per second, so it takes around an hour per file.

Forteen Stars

Re: tFTPGet - Super Slow

hi,

 

there are no special settings for this component 

you could test with command line scp (winscp in case of windows), if it will be also faster - then use tSystem for call command line utility

 

 

-----------
Nine Stars

Re: tFTPGet - Super Slow

With winSCP I am getting between 1.5MB and 2MB per second.

Forteen Stars

Re: tFTPGet - Super Slow

as I understand CyberDuck support multithread for downloading (multiple-connections-for-file-transfers )

winscp - not

so you can check by searching command line clients like - https://aria2.github.io/

or test command line version of CyberDuck - https://duck.sh/

 

 

-----------
Nine Stars

Re: tFTPGet - Super Slow

I used tLibraryLoad to load edtftpj-2.0.4.jar then in tJava:

 

((com.enterprisedt.net.ftp.FTPClient)globalMap.get("conn_tFTPConnection_1")).setNetworkBufferSize(65536);
((com.enterprisedt.net.ftp.FTPClient)globalMap.get("conn_tFTPConnection_1")).setTransferBufferSize(65536);

 

Error:

Detail Message: The method setNetworkBufferSize(int) is undefined for the type FTPClient

Forteen Stars

Re: tFTPGet - Super Slow

why you manually load library, if it already used by component?

 

tFTPLib.PNG

 

and why you are use libray from 2009?!!

Version 2.0.4
(13 February 2009)
-----------
Nine Stars

Re: tFTPGet - Super Slow

If I don't I am getting this error:

Detail Message: com.enterprisedt cannot be resolved to a type

 

I downloaded edtftpj-2.5.0 to use with tLLibraryLoad, now I get:

java.lang.ClassCastException: com.jcraft.jsch.ChannelSftp cannot be cast to com.enterprisedt.net.ftp.FTPClient

Forteen Stars

Re: tFTPGet - Super Slow

my question was - why you are use outdated library?

 

in any case starting from begin of the thread, I not sure increasing only this parameter resolve your problem

 

Duck work fast because run several threads in parallel, in description of library I not found nothing about this 

-----------
Nine Stars

Re: tFTPGet - Super Slow

The duck.sh website is blocked by our firewall.

Is this what you are suggesting?
https://blog.cyberduck.io/2015/06/24/scripting-cloud-storage-copy-using-the-command-line-interface-c...

Forteen Stars

Re: tFTPGet - Super Slow

yes, but just as variant

 

search google - multithread FTP command line client and test what better for you

-----------

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

Agile Data lakes & Analytics

Accelerate your data lake projects with an agile approach

Watch