Remove Multiple new Lines from a particular field

One Star

Remove Multiple new Lines from a particular field

Hi,

I am using Talend Big Data 5.4.1 version. In my job I am receiving an input file which I read using a tfileInputDelimited. In one of the columns there are new lines as a result of which the data is being corrupted. All the felds are text enclosed ("") and the they are delimted by "|". The line separator is newline "\n".

I have used the solution to use CSV options as mentioned in topic : 
This is working when I run the job in my local system in the Windows environment. But when I build my job and run it in the unix server this solution does not work. Could you please help me with this.

Thanks 
Seventeen Stars

Re: Remove Multiple new Lines from a particular field

What you describe is just fine and state-of-the-art. Could it be the case the file to the UNIX machine takes a different way as to your Windows machine, though probably some encodings will be corrupted or line feeds?
I am working on a Mac and your use case is quite normal and works fine - of course also under Unix.
One Star

Re: Remove Multiple new Lines from a particular field

I have tried the dos2unix command to convert the file format after I move the file from Windows to Unix server but that too does not help.
Seventeen Stars

Re: Remove Multiple new Lines from a particular field

hi all,

when you say it doesn't work on linux OS, what's the error ?

regards
laurent
One Star

Re: Remove Multiple new Lines from a particular field

It does not throw any error but the record is malformed when the file is read. As a result of the newline the next record is being treated as a new row.

Thanks
Four Stars

Re: Remove Multiple new Lines from a particular field

Hi,

If line separator is \n and you also have those in between the columns in the same row, I would suggest you to pre-process a file to replace "\n" by some other random character and process file again. You can restore it once the file is processed.
Vaibhav

2019 GARTNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

Have you checked out Talend’s 2019 Summer release yet?

Find out about Talend's 2019 Summer release

Blog

Talend Summer 2019 – What’s New?

Talend continues to revolutionize how businesses leverage speed and manage scale

Watch Now

6 Ways to Start Utilizing Machine Learning with Amazon We Services and Talend

Look at6 ways to start utilizing Machine Learning with Amazon We Services and Talend

Blog