Five Stars

utf8mb4 configuration tmysql

Hi, 

 

I have been trying for days to insert data such smileys, data from all languages into mysql through talend.

 

The source is sql server and target mysql.

 

If I use the 'Set Names utf8mb4' in mysql and run my insert statements, the data is successfully inserted. (Ex - ハウス, is inserted correctly)

 

However, if I set the  'Set Names utf8mb4'  in additional parameters, it just does not work (ハウス is inserted as ???)

 

As of now, its tmssqlinput - tmysqlbulkoutputexec

 

Any idea on how to solve this?

12 REPLIES
Eight Stars

Re: utf8mb4 configuration tmysql

Hi.

 

I am not sure if I am correct, but By default in Talend is used UTF-8 not UTF-8MB4. I think. you need to convert the text to UTF8MB4.

See e.g. this thread:

https://community.talend.com/t5/Design-and-Development/Encoding-UTF-8-and-ISO-8859/td-p/23158

 

Hope it helps.

 

Regards

Lojdr

Five Stars

Re: utf8mb4 configuration tmysql

Yes Lojdr. That's my question precisely. Where do I put the statement 'set names utf8mb4'
Eight Stars

Re: utf8mb4 configuration tmysql

"set names utf8mb4" is related to the connection so in Additional JDBC Parameters as character_set_connection.

 

Five Stars

Re: utf8mb4 configuration tmysql

So, in the additional setting I must include character_set_connection = utf8mb4. What about the other character settings for MySQL which must be set to utf8mb4?

Eight Stars

Re: utf8mb4 configuration tmysql

Character set for table definition or at least for the specific column.

Five Stars

Re: utf8mb4 configuration tmysql

Hi, this does not work for me until I set the character connection ,server connection to utf8mb4.

The tables and columns are already in utf8mb4

So , I am using the set names utf8mb4, which I am unable to use in taken

Eight Stars

Re: utf8mb4 configuration tmysql

Have you converted the string you are sending to the database to the URF8MB4 encoding?

Five Stars

Re: utf8mb4 configuration tmysql

I'm using tmssqlinput. How do I convert the string here to utf8mb4. In the additional parameters of tmssqlinput I have included character encoding = utf8mb4

 

And, also in tmysqlputput I have included in the additional parameters set names utf8mb4

 

But, these don't work when the incoming string has a emoji

Eight Stars

Re: utf8mb4 configuration tmysql

I gave you a link to an example how to convert strings from one encoding to another in Talend in my first post. Sorry, I can't help you more.

Five Stars

Re: utf8mb4 configuration tmysql

I have gone through the link you have sent me in your previous posts lojdr. But, I couldn't find any solution there

 

Anyways, thank you for posting your thoughts

Eight Stars

Re: utf8mb4 configuration tmysql

There is a post from @rhall_2_0 where he showed how to transform text from one encoding to another with tJava. Use that.

Five Stars

Re: utf8mb4 configuration tmysql

The problem was with MySQL version. The tmysqlbulkoutputexec works for mysql 5.6 but not the same with 5.7

 

I guess the handling of utf8 in  tmysqlbulkoutputexec is different for mysql 5.6 and 5.7