Spark Job using a tHMapFile component adds an extra newline character in the output file

Problem Description

The output file generated by the tHMapFile component in Spark batch Job has an extra newline character after every record.

 

Root Cause

Creating a Flat representation adds an __osdtTerminator element at the end of the file. Also, in a Big Data environment, each output record is seen as its own file by the Flat representation. As a result, you get an extra newline character.

 

Solution

Remove the __osdtTerminatorelement from the file representation and clear the Output as delimited? check box in Flat representation.

flat_representation.PNG

Version history
Revision #:
6 of 6
Last update:
‎04-13-2019 12:56 PM
Updated by: