Error useing tRest Component due to utf8

[Anonymous]

Hi
I need to get all tweets from a twitter page. 
The problem is that while twitter use a particular uft8 code, which use more then 2 bytes for save the emoji.
So when I try to read it with Talend it save all this symbols with double "??"
Is there a way to get all the text message correct?

Thanks!

[Anonymous]

If Twitter uses a modified char set you have to use this charset also. One way is building a Talend routine to translate the none-utf8 characters into something useful.

[Anonymous]

Well I use a tRest component to get data from Twitter.
is there a way to set a particular charset in the tREST component?
Tks 🙂

[Anonymous]

Would it be possible to attach the REST component to a java component that would then make this conversion? Or the encoding errors already happen when the tweets are still within the REST component?

[Anonymous]

The error occours in the response (Body) of the tRest component 😞

[Anonymous]

I tried it on a tRESTRequest --(row)-- tLogRow setup and although it was showing them as bars, I could paste them in this reply as an emoji. (But soon after I posted this comment, they became ?? as well)

Input: testing?? (2 emojis, comment changed them into ??)
Output: testing?? (2 emojis, comment changed them into ??)
Screenshot of Job and output:


How did you confirm that the symbols become double "??"? I think if you paste them into a capable editor, you can see the emojis still.

[Anonymous]

Great!
The error was on the last step, when the output is saved in the DbTable
Now I'm trying to save the data correctly...

[Anonymous]

mmm
well, the problem is now on saving the data.

in this flow, the emoji are passed correctly till the last tMap-Output
but when data is saved on the talbe (wich is utf8mb4 encoded) they lost all the emoji and have "double "??" instead...
does enybody know how to solve this?
Tks!

[Anonymous]

risolto in questo modo
uso la funzione: row_originale.testo.getBytes() 
e mappo la colonna del db come byte[]  (pur essendo varchar)