topic Compression does NOT work for Json message for kafka target in Qlik Replicate

Compression does NOT work for Json message for kafka target

RichJ — Sat, 12 Feb 2022 00:45:42 GMT

We need to replicate a table from MS SQL DB to confluent Kafka in Azure. The table has only a few columns but one is CLOB type with max size 1MB. For the kafka target, I chose gzip for Json messages, and "Limit LOB size to (KB) = 300". It did not work out, however, the CLOB column on Kafka site only received 300K data (after un-compression).

It appears that gzip did not work; and neither snappy.

Thanks for help,

Richard

Re: Compression does NOT work for Json message for kafka target

SwathiPulagam — Sat, 12 Feb 2022 02:55:51 GMT

Hi @RichJ ,

If the max CLOB column size is 1 MB then you have to "Limit LOB size to (KB) = 1024". Please try this and let us know how it goes.

Regards,

Swathi

Re: Compression does NOT work for Json message for kafka target

Heinvandenheuvel — Sun, 13 Feb 2022 01:35:25 GMT

That "Limit LOB size to (KB) = 300" is applied to the source reader. That's where the truncation happened, well before any compression.

hth,

Hein