I'm trying to process EDI records using Talend Data Mapper. Their are thousands of records stored in a flat file, with one row per record, delimited by \n. I'm passing the whole file as input to the mapper, and mapping the records to a delimited file. I'm getting this error:
End of file was found in the middle of an EDI segment/display
There are thousands of these errors, and the result is that no records actually get mapped. I'm not sure what to do. Can you send a file with multiple records to the mapper, or do you need to split it into one file per record first (and how would you do that)?
Unfortunately, no, as this is health data. I think I know what is happening, though: I've been sending all the EDI records through the mapper in one large file, and I think when it encounters a bad record, it logs the error, but then keeps reading from that spot. This would explain the cascading failures, since all the records after the bad one would be out of sequence.
When I split the data file so that the mapper only reads one record at a time, I was able to map about 75%, which is consistent with what I know about this data. I did that with tFileOutputDelimited, which lets you specify how many lines you want in each output file. There's probably a more elegant way, but this is working for me for now.