SqlState: 22000 NativeError: 100144 Message: Invalid UTF8 detected in string

[nm21]

Hi Team,

I am getting the error "RetCode: SQL_ERROR SqlState: 22000 NativeError: 100144 Message: Invalid UTF8 detected in string '1000xEF'File 'a015ffd4_cef8_b042_a6c3_9230a7b23386/0/CDC00000001.csv.gz', line 23230, character 1210" in Qlik.

Could you please guide to resolve this error?

Thanks in advance.

Regards,

Nagamanickam K

[mpc]

Hi, 

I think you need to change file encoding to UTF-16: Related post on SO: Snowflake "Invalid UTF8 detected in string SNOWFLAKE" using PUT - Stack Overflow

Regards

[sureshkumar]

Hello @nm21 

For the error

1)Go to Task settings->character substitution->character set->UTF-8 to U+0
2) Reload the task again

If not worked, We do have other options at snowflake end as well

There are two parameters on the Snowflake end that we may also look at.

REPLACE_INVALID_CHARACTERS and IGNORE_UTF8_ERRORS

https://docs.snowflake.com/en/sql-reference/sql/create-file-format.html

REPLACE_INVALID_CHARACTERS will change the encoding to UTF8 and use the UTF8 version of the non UTF8 string.

IGNORE_UTF8_ERRORS is something that should be tested on a lower environment first, as we don't know what Snowflake might put into the database while ignoring the character case.

Regards,

Suresh