Compression does NOT work for Json message for kafka target

[RichJ]

We need to replicate a table from MS SQL DB to confluent Kafka in Azure.  The table has only a few columns but  one  is CLOB type with max size 1MB.  For the kafka target,  I chose gzip for Json messages,  and "Limit LOB size to (KB) = 300".  It did not work out, however,  the  CLOB column on Kafka site only received 300K data (after un-compression).

It appears that gzip did not work; and neither snappy. 

Thanks for help,

Richard

[SwathiPulagam]

Hi @RichJ ,

If the max CLOB column size is 1 MB then you have to "Limit LOB size to (KB) = 1024". Please try this and let us know how it goes.

Regards,

Swathi

[Heinvandenheuvel]

That "Limit LOB size to (KB) = 300" is applied to the source reader.  That's where the truncation happened, well before any compression.

hth,

Hein