[resolved] Wait for Multiple Component Oks

One Star

[resolved] Wait for Multiple Component Oks

I am working on a job where I create a file, process it, and then I want to delete it. I want to handle the potential for reprocessing a file, so I perform a lookup to see if the record exists and then insert/update accordingly. I am iterating over a list of files that are created. So if there are 5 files, I want to process each one individually and after it is done processing, I want to delete it. The problem that I am having is that the delete step is executed after the first ComponentOk hits it and not the last. I tested it with using the "Insert or Update" and "Update or Insert", but the performance was not as good. It took almost twice as long to complete. I thought about moving the delete until after the subjob finishes, but in the case of an error, the files that were successful would not be removed. I appreciate any thoughts on this.

Accepted Solutions
One Star

Re: [resolved] Wait for Multiple Component Oks

Hi,
You could first insert or update the datas and then do a subjob to delete the files which are in the folder.
tfileList-----your job without the deletion
|
on subjob ok
|
tfileList----tfileDelete

Or you could also put your job in an other job. So your job will look like this
tRunJob_1---tFileList----tRunJob_2----oncomponentok----tFileDelete
You would just have to pass the current path of the tFileList in the tRunJob_2.

All Replies
One Star

Re: [resolved] Wait for Multiple Component Oks

Hi,
You could first insert or update the datas and then do a subjob to delete the files which are in the folder.
tfileList-----your job without the deletion
|
on subjob ok
|
tfileList----tfileDelete

Or you could also put your job in an other job. So your job will look like this
tRunJob_1---tFileList----tRunJob_2----oncomponentok----tFileDelete
You would just have to pass the current path of the tFileList in the tRunJob_2.
One Star

Re: [resolved] Wait for Multiple Component Oks

Thank you! The second option worked like a charm. I had thought about the first option, but in the event that one of the files failed, none of them would be removed. The subjob worked perfectly though. Thanks again!

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Best Practices for Using Context Variables with Talend – Part 4

Pick up some tips and tricks with Context Variables

Blog

How Media Organizations Achieved Success with Data Integration

Learn how media organizations have achieved success with Data Integration

Read

APIs for Dummies

View this on-demand webinar about APIs....

Watch Now