topic Re: Error useing tRest Component due to utf8 in Talend Studio

Error useing tRest Component due to utf8

Anonymous — Thu, 03 Dec 2015 17:01:36 GMT

Hi
I need to get all tweets from a twitter page.
The problem is that while twitter use a particular uft8 code, which use more then 2 bytes for save the emoji.
So when I try to read it with Talend it save all this symbols with double "??"
Is there a way to get all the text message correct?

Thanks!

Re: Error useing tRest Component due to utf8

Anonymous — Thu, 03 Dec 2015 19:15:11 GMT

If Twitter uses a modified char set you have to use this charset also. One way is building a Talend routine to translate the none-utf8 characters into something useful.

Re: Error useing tRest Component due to utf8

Anonymous — Fri, 04 Dec 2015 08:00:52 GMT

Well I use a tRest component to get data from Twitter.
is there a way to set a particular charset in the tREST component?
Tks 🙂

Re: Error useing tRest Component due to utf8

Anonymous — Fri, 04 Dec 2015 08:55:31 GMT

Would it be possible to attach the REST component to a java component that would then make this conversion? Or the encoding errors already happen when the tweets are still within the REST component?

Re: Error useing tRest Component due to utf8

Anonymous — Fri, 04 Dec 2015 09:04:50 GMT

The error occours in the response (Body) of the tRest component 😞

Re: Error useing tRest Component due to utf8

Anonymous — Fri, 04 Dec 2015 09:21:22 GMT

I tried it on a tRESTRequest --(row)-- tLogRow setup and although it was showing them as bars, I could paste them in this reply as an emoji. (But soon after I posted this comment, they became ?? as well)

Input: testing?? (2 emojis, comment changed them into ??)
Output: testing?? (2 emojis, comment changed them into ??)
Screenshot of Job and output:

How did you confirm that the symbols become double "??"? I think if you paste them into a capable editor, you can see the emojis still.

Re: Error useing tRest Component due to utf8

Anonymous — Fri, 04 Dec 2015 10:21:38 GMT

Great!
The error was on the last step, when the output is saved in the DbTable
Now I'm trying to save the data correctly...

Re: Error useing tRest Component due to utf8

Anonymous — Fri, 04 Dec 2015 11:35:11 GMT

mmm
well, the problem is now on saving the data.

in this flow, the emoji are passed correctly till the last tMap-Output
but when data is saved on the talbe (wich is utf8mb4 encoded) they lost all the emoji and have "double "??" instead...
does enybody know how to solve this?
Tks!

Re: Error useing tRest Component due to utf8

Anonymous — Wed, 09 Dec 2015 10:55:51 GMT

risolto in questo modo
uso la funzione: row_originale.testo.getBytes()
e mappo la colonna del db come byte[] (pur essendo varchar)