how to increase batch.size and linger.ms for kafka target

[RichJ]

How to increase batch.size and linger.ms for kafka target?

I need to replicate one  SQL Server table that has a CLOB column (LOB limit is set to 200K in the qlik task) to confluent kafka, but got a lot of “rdkafka error: (code=-184) 'Local: Queue full',  and  the table cannot successfully replicate to finish. 

Because in this case, the qlik task is a Kafka producer, I'd like to increase its batch.size and linger.ms or  even buffer.memory.  But how to implement those changes in Qlik replication task?

Thanks for help,

Richard

[lyka]

Hello,

On your kafka target endpoin, you can add the internal parameter rdkafkaProperties
Then set the property value that you need

Example

linger.ms=1000;

You can put in multiple parameters, separated by ;

the list of available parameters are here:
https://github.com/edenhill/librdkafka/blob/v1.3.0/CONFIGURATION.md

Thanks

Lyka

[RichJ · Accepted solution]

Thanks Lyka, it works.