topic Re: [resolved] Encoding issue with tfileOutputDelimited in Talend Studio

[resolved] Encoding issue with tfileOutputDelimited

Anonymous — Wed, 15 Jun 2011 10:03:29 GMT

I use a tfileOutputDelimited with encoding set on UTF-8 (default) in advanced parameters.
Nevertheless the produced file has an UTF-16LE BOM (FF FE) with UTF-16LE character encoding.
I tried to pipe a tChangeFileEncoding (UTF16->UTF8) with and without custom input encoding.
Both tests failed, i'm stuck with utf16.
Any idea ?
Frankie.
BTW : I use TOS Version: 4.1.2
Build id: r53616-20110106-0635

Re: [resolved] Encoding issue with tfileOutputDelimited

Anonymous — Wed, 15 Jun 2011 15:24:37 GMT

Hi
Can you send me an example file for testing?
Best regards
Shong

Re: [resolved] Encoding issue with tfileOutputDelimited

Anonymous — Thu, 16 Jun 2011 09:52:18 GMT

Seems it is related a " Will Not Fix" Java Bug.
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4508058
A quick and dirty workaround (in a tJava) :

//reading as UTF-16LE
FileInputStream fis = new FileInputStream("inpufile.txt");
BufferedReader r = new BufferedReader(new InputStreamReader(fis, "UTF-16LE"));
//writing as UTF-8
FileOutputStream fos = new FileOutputStream("ouputfile.txt");
Writer w = new BufferedWriter(new OutputStreamWriter(fos, "UTF-8"));
//copy data
for (String s = ""; (s = r.readLine()) != null;) {
	w.write(s + System.getProperty("line.separator"));
	w.flush();
}
//closing streams
w.close();
r.close();

The BOM is still wrong, but the encoding is right.
I did not find a convenient way to put binary files online.
So here is a small example of what I mean :
- Actual input data (readable) : ?
(LATIN CAPITAL LETTER A WITH DIAERESIS + DEGREE SIGN)
- Correct UTF-16LE (hexa) : FF FE 00 C4 00 B0
as written by talend in my case (supposed to be utf8)
- Actual output file (hexa mixed) : FF FE C3 84 C2 B0
after the above quick'n dirty conversion
- Expected output (hexa utf8) : EF BB BF C3 84 C2 B0
Edit : Oops seems that ultraedit converts automatically to utf16 when opening. Trying with a decent binary viwer/editor now.

Re: [resolved] Encoding issue with tfileOutputDelimited

Anonymous — Thu, 16 Jun 2011 10:43:38 GMT

Finally... Produced file is UTF-8, as expected, but without BOM.
My bad, the default configuration for my tool (ultraedit32) was to translate file to UTF when recognized as such, showing wrong BOM in my case (and adding it if saved).
I'll set this post as resolved.

Re: [resolved] Encoding issue with tfileOutputDelimited

Anonymous — Thu, 16 Jun 2011 11:18:46 GMT

Hi
Glad to see that you find the cause! Maybe you can try this component tWriteHeaderLineToFileWithBOM to output the records with BOM.

Best regards
Shong