topic Re: Is it possible to detect if a file is in expected encoding (utf8)? in Talend Studio

Is it possible to detect if a file is in expected encoding (utf8)?

olja — Tue, 24 Nov 2020 13:35:39 GMT

I have a java library which consumes incoming files. The problem is, that it is only working if the file is utf8 encoded. Is there a component or a best practice to check this with talend? I want to reject files which are not utf8 encoded.

Thanks

Re: Is it possible to detect if a file is in expected encoding (utf8)?

Anonymous — Wed, 25 Nov 2020 09:12:52 GMT

Hello,

So far, there is no a component or a built-in function can be used to detect the file encoding. You could write a routine in Talend to parse the file encoding.

https://stackoverflow.com/questions/499010/java-how-to-determine-the-correct-charset-encoding-of-a-stream

Best regards

Sabrina

Re: Is it possible to detect if a file is in expected encoding (utf8)?

olja — Mon, 30 Nov 2020 10:22:20 GMT

Thanks for the reply,

Maybe someone has the same problem, i have solved it with an external jar

With that org.mozilla.universalchardet.UniversalDetector

https://github.com/albfernandez/juniversalchardet

it worked quite good. I have added a Java Routine and using it my job, if it is not return true, then I will throw an exception and the file will be handled differently

Re: Is it possible to detect if a file is in expected encoding (utf8)?

Anonymous — Tue, 01 Dec 2020 08:27:02 GMT

Hello,

Great it works. Thanks for sharing it with us on community.

Best regards

Sabrina