File source endpoint - codepage

[Abrie_M]

Good day

I'm using a file source endpoint - normal text file on Windows and need to replicate it to Hadoop (text format).

When I query the column in Hive, it displays the hex value.  Any other replications with the same target endpoint (hadoop), but from e.g. DB2,  works correctly.

"Marais" displays as "4D6172616973"

Any suggestions? I've tried different code pages, but the input file is a normal windows text file, so it should be 65001.

Thanx

Abrie

[john_wang]

@Abrie_M ,

Can you check what's the text values in HDFS files? and what's the data type of the column in HIVE?

It's better you open a support ticket and provide a sample file, and also the task Diag Package so far we may reproduce it in our labs to understand it correctly.

Regards,

John.

[Abrie_M]

Hi John

Thanx for the reply. The values are the same in the csv file on HDFS.

I will log a support ticket.

Regards

Abrie

[john_wang · Accepted solution]

Hello @Abrie_M ,

From JSON file you defined the NAME column as BYTES

this request Replicate to read the column as-is (by default the code page is 65001). Please change the data type from BYTES(50) to STRING(50) in source table definition Type.

Let me know if it works for you.

Regards,

John.

[Abrie_M]

Thanx John! I've changed it to string as you suggested and the data in our lake is correct now.

Regards

Abrie

[john_wang]

Hello @Abrie_M ,

Glad to hear that and Thanks for the feedback.

Best Regards,

John.