topic Re: File source endpoint - codepage in Qlik Replicate

File source endpoint - codepage

Abrie_M — Fri, 19 Aug 2022 08:11:23 GMT

Good day

I'm using a file source endpoint - normal text file on Windows and need to replicate it to Hadoop (text format).

When I query the column in Hive, it displays the hex value. Any other replications with the same target endpoint (hadoop), but from e.g. DB2, works correctly.

"Marais" displays as "4D6172616973"

Any suggestions? I've tried different code pages, but the input file is a normal windows text file, so it should be 65001.

Thanx

Abrie

Re: File source endpoint - codepage

john_wang — Fri, 19 Aug 2022 08:37:55 GMT

@Abrie_M ,

Can you check what's the text values in HDFS files? and what's the data type of the column in HIVE?

It's better you open a support ticket and provide a sample file, and also the task Diag Package so far we may reproduce it in our labs to understand it correctly.

Regards,

John.

Re: File source endpoint - codepage

Abrie_M — Fri, 19 Aug 2022 08:52:18 GMT

Hi John

Thanx for the reply. The values are the same in the csv file on HDFS.

I will log a support ticket.

Regards

Abrie

Re: File source endpoint - codepage

john_wang — Sat, 20 Aug 2022 02:21:10 GMT

Hello @Abrie_M ,

From JSON file you defined the NAME column as BYTES

this request Replicate to read the column as-is (by default the code page is 65001). Please change the data type from BYTES(50) to STRING(50) in source table definition Type.

Let me know if it works for you.

Regards,

John.

Re: File source endpoint - codepage

Abrie_M — Mon, 22 Aug 2022 07:29:04 GMT

Thanx John! I've changed it to string as you suggested and the data in our lake is correct now.

Regards

Abrie

Re: File source endpoint - codepage

john_wang — Mon, 22 Aug 2022 07:40:01 GMT

Hello @Abrie_M ,

Glad to hear that and Thanks for the feedback.

Best Regards,

John.