tFileList vs tWaitForFile for large files in transfer

One Star

tFileList vs tWaitForFile for large files in transfer

Hi,
I have one case with large files say about 2GB transfer with sftp and doing further process and finally load into db.
1. one full time runnnig job with tFTPGet to get files and input into */input.
2. one full time running job with tWaitForFile monitor */input and doing validation and filter operation for each file coming to this directory. and moved processed file into */archive folder
The current issue is that for large file, that has not finished transfer in job 1, job 2 started processing the file, which means the file processed is incomplete and caused only part of data loaded into db.
Does anybody have idea on how to handle this?

And I have done a similar test as below
1.one job generate large files about 2GB/each file into */input
2.one job tWaitForFile monitor */inpout and just copy this file into another */copy direcotry.
3.one job tFileList retrieve files under */input and just copy this file into another */copy direcotry.
The result is
when the file has not finished writing say a.txt in */input
either job2 or job3 will start copy a.txt into */copy, and eventually the file size copy in */copy is not the same as in */input.
Though have set 'Wait for file to be release', still the same.
How could we design this job to avoid this incomplete file case.
One Star

Re: tFileList vs tWaitForFile for large files in transfer

Hi,
Create a topic in the bugtracker, it's not normal that it's not possible with the twaitforfile component.

Re: tFileList vs tWaitForFile for large files in transfer

hey mints ..
did you get this resolved??

Calling Talend Open Studio Users

The first 100 community members completing the Open Studio survey win a $10 gift voucher.

Start the survey

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

APIs for Dummies

View this on-demand webinar about APIs....

Watch Now