Slow FTP upload because of files listing: any workaround?

Nine Stars

Slow FTP upload because of files listing: any workaround?

Hi all,

We have some jobs using Talend to upload files on FTP that have always been slow but we never took the time to investigate until today.

I saw (in the generated code as well as with Wireshark) that each time the tFTPPut component is called, it first starts with a directory listing.
That's a problem for us as the FTP destination directory is large with a lot of files. In consequence, for each uploaded file (50 kB for the biggest, 5kB for the smallest), there is a directory listing that took approximately 500 kB (between 10 and 100 times the size of the file send).

Any workaround to avoid a full directory listing for each file?

Tested on a very simple job:
tPreJob->TFTPConnection
tFileList->tFTPPut (using existing connection)

Thanks.
Regards.

Alexis

Community Manager

Re: Slow FTP upload because of files listing: any workaround?

Hi
What's the type of overwrite file on your tFTPPut component?
----------------------------------------------------------
Talend | Data Agility for Modern Business
Nine Stars

Re: Slow FTP upload because of files listing: any workaround?

Hi,

Overwrite is set on "Always"

Regards

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 1

Learn how to do cool things with Context Variables

Blog

Migrate Data from one Database to another with one Job using the Dynamic Schema

Find out how to migrate from one database to another using the Dynamic schema

Blog

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog