topic Re: [resolved] Encoding Problem with Special Characters in Talend Studio

[resolved] Encoding Problem with Special Characters

Anonymous — Wed, 26 Jun 2013 12:58:24 GMT

Hi all,
we're using TOS DI 5.2.0M3 and facing a problem mapping data from mssql to oracle (and even within the same mssql database) when the target field has a smaller size than the source field (varchar 132 to varchar 40) AND it contains special chars like Ü,Ä etc..
The String is cut to a length of 40 chars in a tmap component but a data truncation error occurs when saving this String using a tXXXOutput Component.
Does anyone have a solution where an encoding can be set in the tXXXOutput Components or any information how this problem can be solved?
Thanks in advance,
M.Meier

Re: [resolved] Encoding Problem with Special Characters

Anonymous — Thu, 27 Jun 2013 08:23:32 GMT

Hi,
Perhaps this KB article about Exception: Data truncation: Data too long for column will be helpful for you.
Best regards
Sabrina

Re: [resolved] Encoding Problem with Special Characters

Anonymous — Thu, 27 Jun 2013 09:00:46 GMT

Hi xdshi,
thanks for your hint. I tried this, but got the same error. What i did to solve the problem was just to remove the selection of "die on error" on my output component. I now get the message "data truncation" in the log, but it commits the data completely anyway....
Regards,
M.Meier

Re: [resolved] Encoding Problem with Special Characters

Anonymous — Thu, 27 Jun 2013 09:29:23 GMT

Hi,
The special characters shows well in target DB? There is no error in console but the error info "data truncation" in log?
Could you give us your job screenshot in forum which may shed light to us.
Best regards
Sabrina

Re: [resolved] Encoding Problem with Special Characters

Anonymous — Fri, 28 Jun 2013 13:07:11 GMT

Hi xdshi,
i did some further investigation, it pointed out i made a mistake on the mssql -> mssql mapping, the string had too much characters so the error message was correct.
Concerning mssql -> oracle it was a bit trickier. The destination oracle database (i had no influence on the schema) has varchar fields with exactly the same size as the source (i thought ) but the was a small difference is saw when i inspected the oracle schema:
the definition of the fields was varchar (254 BYTE) which means the field has space for 254 BYTE. Since we are working with multibyte characters (ÄÖÜ etc.) we need 4 bytes space for saving it in the database.
So my solution is to count the occurences of these multibyte chars in the source field and cut the string (len - 4*occurences). I'm sure it's not the bust way, but it works for now
Regards,
M

Re: [resolved] Encoding Problem with Special Characters

Anonymous — Mon, 01 Jul 2013 04:43:47 GMT

Hi,
Thanks for your feedback and sharing your experience with us, I will make some investigation to find a better way for your issue.
Best regards
Sabrina

Re: [resolved] Encoding Problem with Special Characters

Anonymous — Fri, 05 Jul 2013 10:30:42 GMT

Hi xdshi,
i think the problem in my case is a bad schema design, which i sadly can't change. But it might be a cool feature if the encoding can be set in the output elements, although i know encoding handling using java Strings is a bit tricky
Regards,
M

Re: [resolved] Encoding Problem with Special Characters

Anonymous — Fri, 05 Jul 2013 10:51:01 GMT

Hi,
Thanks your response.
In addition, For talend enterprise subscription version, there is "Dynamic schema" feature which allows you to design schema with an unknown column structure (unknown name and number of columns).I think it is a good option for "Schema" Issue.
You can refer to this article How to process changing data structure.
Best regards
Sabrina