topic Re: Special characters in Talend Studio

Special characters

Anonymous — Mon, 24 Jul 2017 14:50:07 GMT

Hi,

I have a Delimited file with an UTF-8 encodage, but I have few characters as "é", "à"... wich are a black diamond with the character ? inside.

And a can't choose this case as condition in a tFilterRow.

Do you have an idea to resolve this issue ? Can you help me, please?

Best regards

Re: Special characters

Anonymous — Mon, 24 Jul 2017 14:57:43 GMT

Are you needing to get rid of the characters, or select based on them? I think Talend has a function to strip non-standard characters from text strings. To select on them, try using \uxxx and the character's Unicode value.

David

Re: Special characters

vboppudi — Mon, 24 Jul 2017 17:24:50 GMT

Hi,

Please provide some sample source data and expected output.

Regards,

Re: Special characters

Anonymous — Mon, 24 Jul 2017 18:57:14 GMT

I don't have time to do that, but if you create a tMap component and look at the functions available under String Handling, you'll see what I'm referring to.

David

Re: Special characters

TRF — Tue, 25 Jul 2017 08:59:17 GMT

Hi,
Are you sure the source file is UTF-8 encoded?
When do see the "black diamond with the character ? inside"?

Can you share an extract of the input file?

Re: Special characters

Anonymous — Tue, 25 Jul 2017 12:36:23 GMT

Hello,

This is my job :

tFileList_1 :

directory : "D:/SPC/FichienSensibilite"

files : "*.dat"

tFileInputDelimited_1 :

flow's name :

((Strint)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

tFilterRow_1 :

condition

Freq Vide Vaut "Fréquence=1000Hz"

Run IF :

condition :

((Integer)globalMap.get("tFilterRow_1_NB_LINE_OK")) > 0

tFileCopy_1 :

file's name :

((String)globalMap.get("tFile_1_CURRENT_FILEPATH"))

directory : "D:/SPC/test"

For my tFileInputDelimited_1 component, I use a metadata delimited file wich is form :

Colonne0

Date=20151202

Fréquence=1000Hz

Temps=50µs

Information

Colonne0 is a String type.

I want to search "Fréquence=1000Hz" and "Temps=50µs" in the tFilterRow_1 component, but "é" and "µ" are change by black diamonds with the character ? inside in my metadata preview.

My encode is UTF-8.

Best regards

BastienM

PS : My job is in French

Re: Special characters

TRF — Tue, 25 Jul 2017 13:07:55 GMT

Once again, are you sure of the file encoding?

Can you open the file using Notepad++ then, menu Encode and you should have something like that:

I just made the test with a CSV file with this encoding, the preview is OK.

If I change the file encoding to ANSI (using Notepad++) and save the file, preview is not OK as you can see here:

Hope this helps.

Re: Special characters

Anonymous — Tue, 25 Jul 2017 13:25:17 GMT

Hi TRF,

It's good for my metatdata, it was encode in ANSI so I changed its. Now I'm a new issue, it's that all input files are encode in ANSI, so what exit a solution to change all files in UTF-8 encode with an automatic process?

Best regards

BastienM

Re: Special characters

TRF — Tue, 25 Jul 2017 13:33:01 GMT

No, you can just forget the preview (it's just a preview!) and set the Encoding as ISO-8859-15 in tFileInputDelimited (Advanced settings).

Should work.

Re: Special characters

Anonymous — Tue, 25 Jul 2017 14:06:20 GMT

Hi,

It's a success, thank you.

Best regards

BastienM