tFileList vs tWaitForFile for large files in transfer
Hi, I have one case with large files say about 2GB transfer with sftp and doing further process and finally load into db. 1. one full time runnnig job with tFTPGet to get files and input into */input. 2. one full time running job with tWaitForFile monitor */input and doing validation and filter operation for each file coming to this directory. and moved processed file into */archive folder The current issue is that for large file, that has not finished transfer in job 1, job 2 started processing the file, which means the file processed is incomplete and caused only part of data loaded into db. Does anybody have idea on how to handle this?
And I have done a similar test as below 1.one job generate large files about 2GB/each file into */input 2.one job tWaitForFile monitor */inpout and just copy this file into another */copy direcotry. 3.one job tFileList retrieve files under */input and just copy this file into another */copy direcotry. The result is when the file has not finished writing say a.txt in */input either job2 or job3 will start copy a.txt into */copy, and eventually the file size copy in */copy is not the same as in */input. Though have set 'Wait for file to be release', still the same. How could we design this job to avoid this incomplete file case.