Removing Special Characters

Highlighted
Six Stars

Removing Special Characters

Hi All,

 

I am currently trying to import data from a csv file with Chinese Characters and there are some � in the csv file..

 

My question is how do I import the data into teradata without it giving the "The string contains an untranslatable character," error message.

 

I would like to either remove the � or just replace it with an empty space.

 

Please help!

 

Thanks in advance.

 

 


Accepted Solutions
Employee

Re: Removing Special Characters

Ok. In that case, could you please go to the advanced settings of the tfileinput component and convert the language settings to UTF-8.

 

The symbols are getting generated due to wrong language set. You can remove the tMap in this case.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)


All Replies
Employee

Re: Removing Special Characters

Hi,

 

    You can remove all non-Latin characters by using below function in a tmap.

 

row2.input_data.replaceAll("[^\\x00-\\x7F]", "")

image.png

 

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

Six Stars

Re: Removing Special Characters

Hi Nikhil,

 

But if that's the case, my chinese character words (such as 小姐) will also be an empty string right?

I just want to remove those garbled letters such as ...  '' 

 

thanks.

 

 

Employee

Re: Removing Special Characters

Ok. In that case, could you please go to the advanced settings of the tfileinput component and convert the language settings to UTF-8.

 

The symbols are getting generated due to wrong language set. You can remove the tMap in this case.

 

Warm Regards,
Nikhil Thampi

Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved :-)

15TH OCTOBER, COUNTY HALL, LONDON

Join us at the Community Lounge.

Register Now

2019 GARNER MAGIC QUADRANT FOR DATA INTEGRATION TOOL

Talend named a Leader.

Get your copy

OPEN STUDIO FOR DATA INTEGRATION

Kickstart your first data integration and ETL projects.

Download now

What’s New for Talend Summer ’19

Watch the recorded webinar!

Watch Now

Put Massive Amounts of Data to Work

Learn how to make your data more available, reduce costs and cut your build time

Watch Now

How OTTO Utilizes Big Data to Deliver Personalized Experiences

Read about OTTO's experiences with Big Data and Personalized Experiences

Blog

Talend Integration with Databricks

Take a look at this video about Talend Integration with Databricks

Watch Now