Loading data with Polish characters from QVD these are gone

[flaurntiu]

Currently we have a customer that uses Polish characters in their SQL database. These are shown in the result set when running SQL Select statements.

When using the same select directly in Qlik Sense, the characters are shown in Qlik Sense apps.

When the data is loaded from SQL in a Qlik sense App and then written to a QVD file, the Polish characters seem to be gone. Retrieving the QVD contents using Q-Eye will not show these characters. Loading data from the QVD into a Qlik Sense app does not show the characters.

I have seen this remotely and am waiting on the customer SQL database to be uploaded for further testing.

Does anyone know how to fix this?

[flaurntiu]

Could this have something to do with issue QLIK-67044?
The customer is now using Qlik Sense 3.1.x

[flaurntiu · Accepted solution]

I seem to have found the cause for this.
When loading the data from the SQL server, it can be in several code pages. When the SQL field contains special characters it will load them properly when read from the database. However if the data is not in Unicode format (e.g. nvarchar, nchar, etc.)  the data will not be stored properly into a QVD file. The special characters will be substituted by a non-Unicode counterpart.This means that the character 'Ł' will be replaced by 'L', 'ą' by 'a' and so on.

Therefore you will need to rewrite the SQL queries to cast or convert the text fields to Unicode format.

An example:

This results in the following QVD contents (part of) for the Accounts1 table without Unicode:

This results in the following QVD contents (part of) for the Accounts2 table with Unicode:

As you can see the second file keeps the special characters, as they are recognized as unicode characters and stored properly in the QVD.
Hope this helps others running into this issue as well.
Have fun Qlik-ing around