topic Re: [resolved] tFileInputDelimited not reading international characters (UTF-8) in Talend Studio

[resolved] tFileInputDelimited not reading international characters (UTF-8)

Anonymous — Sat, 16 Nov 2024 12:26:32 GMT

I am reading utf-8 encoded CSV text files, but am getting errors when reading the file with tFileInputdelimited. Once this is working, these will be saved (via a tmap -> tOracleOutput), to Oracle 11g. I am not sure if I then need to set advanced options on the tOracleOutput. The oracle db has been configured to store muti-byte characters.
Probably something simple I am missing.
I have attached screenshots.
Dave

Re: [resolved] tFileInputDelimited not reading international characters (UTF-8)

Anonymous — Thu, 19 Jan 2012 05:17:56 GMT

Hi
Try this. Set Encoding "Custom"->"GBK".
Regards,
Pedro

Re: [resolved] tFileInputDelimited not reading international characters (UTF-8)

Anonymous — Thu, 19 Jan 2012 19:31:29 GMT

thanks - I am investigating another issue with this. If that does not work, I will definitely try this. In any case, I will keep this post updated.
Thanks!
Dave

Re: [resolved] tFileInputDelimited not reading international characters (UTF-8)

Anonymous — Fri, 20 Jan 2012 21:58:47 GMT

"GBK" did not work. I have escalated this to Talend support.
thanks,
Dave

Re: [resolved] tFileInputDelimited not reading international characters (UTF-8)

Anonymous — Sat, 21 Jan 2012 01:22:29 GMT

Hi Dave
From the error message, we can see that it is a Number Format exception throws on tFileInputDelimited_2, one of columns is read using Integer/int data type. Try to change it to string data type.
Best regards
Shong

Re: [resolved] tFileInputDelimited not reading international characters (UTF-8)

Anonymous — Mon, 23 Jan 2012 23:50:57 GMT

Shong,
Well, the first column is a short (Integer).
I changed the first column to a string in the FileInput, and added a tConvertType, after the FileInput. In the tConvertType, I convert the first column from string to short.
I now get a new error (new "Convert" "screenshots attached)
Dave

Re: [resolved] tFileInputDelimited not reading international characters (UTF-8)

janhess — Tue, 24 Jan 2012 09:29:09 GMT

Have you tried giving your ACD_No a size?

Re: [resolved] tFileInputDelimited not reading international characters (UTF-8)

Anonymous — Tue, 24 Jan 2012 15:28:36 GMT

I can. However, I think the issue however, is that Talend is not resolving UTF8 encoded data. In the screenshots, there are characters that Talend cannot resolve. I struggle with this however, as I cannot find any posts that also share this problem.

Re: [resolved] tFileInputDelimited not reading international characters (UTF-8)

Anonymous — Wed, 25 Jan 2012 22:34:22 GMT

So, here is what I did to resolve:
I brought up the source file in Firefox (file open). Then rt-click - View page info. This shows the character encoding. I saw that this file was actually encoded as UTF-16, not UTF-8, as I ad thought. I then changed the tFileinput to custom "UTF-16" in Talend, and it worked fine.
Dave