topic Re: tChangeFileEncoding and UTF8 encoding in Talend Studio

tChangeFileEncoding and UTF8 encoding

Anonymous — Mon, 07 May 2018 10:53:47 GMT

Hello,

I have an input_file encoded in ANSI that I want to encode to UTF-8.

So basically, I use the tChangeFileEncoding component and I do get an output_file encoded in UTF-8. While I open it with notepad++, everything is alright.

But when i open it with Excel, "€" and "é" caracters show me things like "â‚¬_" and "Ã©".

Is there any way to fix this ?

Re: tChangeFileEncoding and UTF8 encoding

manodwhb — Mon, 07 May 2018 10:58:18 GMT

@JoshyBrown,what type of file are you using to change the coding?

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Mon, 07 May 2018 11:00:54 GMT

It's a .csv file.

Re: tChangeFileEncoding and UTF8 encoding

manodwhb — Mon, 07 May 2018 11:22:19 GMT

@JoshyBrown,based on the encoding those characters will be converted into special characters.

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Mon, 07 May 2018 11:47:20 GMT

@manodwhb, Is there a way to change/by pass that and obtain a proper .csv file when opened with excel ?

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Wed, 09 May 2018 14:17:40 GMT

I started to get a grasp on your awnser and the solution to fix my problem is to use the BOM. ~~Unfortunately, while using tFileChangeEncoding and indicating "UTF-8-BOM", Talend can not recognize it and therefore deliver a proper output file.~~

~~Anyone knows how to use the BOM in Talend ? Or use the custom encoding option ?~~

*edit*

Ok, it's not how it works. I have found this topic which is related to my problem. Apparently, I need to use a custom component in order to use BOM. BOM is not native on Talend. But maybe the previous topic is too old. I can't find the tWriteHeaderLineToFileWithBOM component. Is there a way to download it or did the OP retrieve it ?

The key to my problem is the BOM. I'm sure of it. Once I can download, install and use that custom component, my problem will be solved.

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Fri, 11 May 2018 04:49:05 GMT

Hello,

Could you please refer to this link about:https://exchange.talend.com/#marketplaceproductoverview:marketplace=marketplace%252F1&p=marketplace%252F1%252Fproducts%252F257&pi=marketplace%252F1%252Fproducts%252F257%252Fitems%252F363?

And feel free to let us know if you can download this custom component from talend exchange portal.

Best regards

Sabrina

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Tue, 29 May 2018 11:03:12 GMT

If anyone ever has the same problem, here is how I solved mine. As a reminder, I needed to change the encoding of a file.csv from ANSI to UTF-8. And I also had a problem with my UTF-8 file when I opened it with Excel.

First things first, it is apparently well known that excel has trouble dealing with files.csv in UTF-8. (example here). And since, the file didn't have to be used in Excel in the end, I just ignore that part.

Secondly, I found that my file was not encoded in ISO-8859-15 (aka Latin-9) as I thought it was natively but in Latin-1. I tried using the option "Custom" encoding from the tFileChangeEncoding to do the job, but it was not as intuitive as I thought it would be. So I used a tJava component + a custom routine to solve this problem. For the routine, I used the java.NIO library and I found here all the encoding supported by this library. My encoding is/was "windows-1252".

After that, I simply had to call my routine like :

myPackage.MyCustomRoutine.myMethod (input_encoding, output_encoding, input_directory+input_filename, output_directory+output_filename);

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Sun, 27 Jan 2019 09:24:40 GMT

Hello Joshy,

Please, can you share your routine please ?

Thanks you !!

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Mon, 28 Jan 2019 08:51:36 GMT

Hello @sasafca ,

You'll find it in the join piece to this message.

Hoping it will help.

routine_encoding.txt

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Tue, 29 Jan 2019 10:37:01 GMT

Hello, @JoshyBrown

You are a Genius !!

Thanks you so much !

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Tue, 29 Jan 2019 10:40:15 GMT

I'll take the compliment even though this problem gave me quite a hard time

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Tue, 29 Jan 2019 13:04:39 GMT

@JoshyBrown, do you know how to convert CSV UTF-8 to UTF-8-BOM please ?

My client need a CSV in UTF-8... with BOM for special caraters... (not "windows-1252" regrettably)

Thanks you

Re: tChangeFileEncoding and UTF8 encoding

Anonymous — Tue, 29 Jan 2019 13:25:00 GMT

Sorry, that I don't know. I didn't have to go through this problem.

Re: tChangeFileEncoding and UTF8 encoding

CDol1613983942 — Tue, 24 May 2022 05:35:43 GMT

Hi @David Joshua ,

Can you please give details for tjava component custom routine?