One Star

How to delete row from a delimited file?

Hello,
I have a tFileList to iterate tFileInputDelimited files.
For each file, I check row.data. If it's wrong (tMap filter) I copy the wrong row to another file for further processing.
At that time I would like to delete this wrong row from the file I'm still reading (row iteration). How can I do it with TOS?
How can I Manage Read/Delete operation on a delimited file inside the same job?
4 REPLIES
One Star

Re: How to delete row from a delimited file?

I think you should use a tFilterRow instead of a tMap.
Have you tried :
tFileList1 -(iterate)-> tFileInputDelimited1 -(main)-> tFilterRow1 -(filter)-> tFileOutputDelimited1
-(reject)-> tFileOutputDelimited2
And in tFilterRow parameters you specify the condition on your "data" column.
But it might have misunderstood your problem.
One Star

Re: How to delete row from a delimited file?

In the tFilter component, filter function call is limited to existing ones(Match, length, ...). In my tMap filter I use a personnal function that does a scan in one of the String data to guess if it's OK or NOT.
Moreover I think the tFileOutput should not be the same as the input one in case you want to append the original file with new rows.
I'm trying to reject wrong row to my new file to be processed and copy the correct rows in another file so that I could delete the older one and rename the new one at the end of the job. But I'm not glad about that solution as an existing application will read the file.
That's why I really want to delete row in the original file before the other application would read it (process failure in case of wrong data).
One Star

Re: How to delete row from a delimited file?

In the tFilter component, you can check the box "use advanced mode" to use your personnal function (I often call methodes in my routines this way). The tMap with its "output reject" option also works fine but I think when you need to filter, you should make it appear graphically by using a tFilter.
About deleting rows from your input file, I don't think this can be done by Talend. Another solution for your problem would be to first rename your file and then generate a file with the name expected by the application.
One Star

Re: How to delete row from a delimited file?

thanks for your remark.
With tFilter or tMap, I got the same results. tFilter is more appropriate in that case.
I realize the only issue would be to rename, copy and replace as you suggested.