topic Re: Convert a CSV UTF-8 to UTF-8-BOM in Talend Studio

Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Sat, 16 Nov 2024 06:42:07 GMT

Hello,

Please i need some help.
I have to output a CSV file in UTF-8-BOM for my client, because in UTF-8 several special caracters are not encoding as well...

In this topic, I found a routine which can encode CSV file, but not in UTF-8-BOM...

https://community.talend.com/t5/Design-and-Development/tChangeFileEncoding-and-UTF8-encoding/m-p/149125#M91288

Please, somebody has a solution to this problem ?

PS: not by passing with the custom component tWriteHeaderLineToFileWithBOM

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 10:05:24 GMT

To do this you will need to add a bit code. I have just tested it, and it appears to work if you test the file here: https://validator.w3.org/i18n-checker/check#validate-by-upload+

The way to do this is to add a tJava to the beginning of your flow (before the file output component). Then add this code (or a variant of this code) to the tJava.....

String filename = "/Users/richardhall/Downloads/test.CSV";
String content="";
byte[] bytes = content.getBytes();
 
try (OutputStream out = new FileOutputStream(filename)) {
 
	// write a byte sequence
	out.write(0xEF);
	out.write(0xBB);
	out.write(0xBF);
        
	out.close();
} catch (IOException e) {
	e.printStackTrace();
}

You will also need to add these imports to the tJava's advanced settings...

import java.io.FileOutputStream;
import java.io.OutputStream;

What this does is create the file and adds 3 BOM bytes to the beginning of the file. The only change you need to make to the file output component is to set the file to Append and to not fail if the file already exists.

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 10:17:57 GMT

@rhall thanks you for your answer !

I don't understand how i can write the file path in the tJava before my file is created ?

String filename = "/Users/richardhall/Downloads/test.CSV";

Do you know how modify this routine and add your code to encode in UTF-8-BOM (if convert selection is UTF-8 for exemple) please ?

Thanks you again !

routine_encoding.txt

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 10:22:51 GMT

Sorry, that was my filepath I left in there 🙂

If you set the filepath in your component with a context variable, just use the same context variable. Otherwise hard code it as I did in my test. Probably best to use a context variable though 🙂

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 11:55:35 GMT

@rhall ok thanks you.
So, I can put my file path (context variable) before the file exists ?

Thanks !!

PS:

I tried but doesn't work... CSV file generate is UTF8 not UTF8-BOM.

Opened with Excel, special caracters are not good (é = Ã©)...

UTF8.JPG

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 12:47:10 GMT

Check the file with the link I provided. That worked for me. I have no other way of testing, so maybe you can tell me how you tested. Run the job without the Java code and test the file with the link I gave. Then run the job with the code switched on and test the file. You should see that the BOM characters are recognised.

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 12:57:19 GMT

@rhall

In my component tFileOutPutDelimited, how type of encoding i have to put ?

Have I to enable "write after" ? please

Thanks !

PS: i have tested, my component tFileOutputDelemited, convert the CSV file UTF-8-BOM to UTF-8...

IT'S WORKED !! I have to enable "write after" in component tFileOutPutDelimited !

@rhall thanks you so much !!

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 13:46:40 GMT

No problem 🙂

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 14:49:41 GMT

@rhall

I have just a last problem, with this solution i can't have the title in my csv...

Can you tell me how add it ?
Maybe in the tJava, like out.write("TitleColumnA; TitleColumnB"); ??

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 16:15:40 GMT

Why can't you add your title? I don't understand?

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 16:32:14 GMT

I don't know why title of my CSV column disapear...

So i changed your tJava code for this code and it's work :

String filename = "test.csv";
String content="";
byte[] bytes = content.getBytes();
 

try (PrintStream w = new PrintStream(new FileOutputStream(filename))) {
	w.write('\ufeef'); // emits 0xef
    w.write('\ufebb'); // emits 0xbb
    w.write('\ufebf'); // emits 0xbf
    w.print("OptionValue,OptionName\n");
} catch (IOException e) {
	e.printStackTrace();
}

Thanks you again @rhall

Re: Convert a CSV UTF-8 to UTF-8-BOM

Anonymous — Fri, 01 Feb 2019 20:28:10 GMT

Oh I see. You raised a question about file name and row count previously didn't you? All you need to do to enable that is to put the tJava before the subjob that writes the header values. But you seem to have sorted this 🙂