How to configure Kafka producer properties?

[David_Fergen]

The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution below may not be provided by Qlik Support.

Description:

Replicate is using librdkafka to produce messages to Kafka. The Kafka producer has a set of configuration properties with default values. These values can be configured using internal parameters.

SOLUTION:

Configuring the Kafka Producer

The list of configuration properties is available at the following URL:
https://github.com/edenhill/librdkafka/blob/2df048ba8118cba73d0551564b7a15eeb65cd5b8/CONFIGURATION.m...

There are two levels of properties: 

1. Global - use internal parameter rdkafkaProperties to configure these properties
2. Topic - use internal parameter rdkafkaTopicProperties to configure these properties

Note:

• Only Producer properties can be configured (marked with “P” or “*”), not Consumer properties (marked with “C”).
• Use a semicolon (;) to specify multiple Global-level properties or multiple Topic-level properties.

Examples:

Configuring the Global-level properties topic.metadata.refresh.interval.ms and socket.timeout.ms 
 

Parameter	Value
rdkafkaProperties	topic.metadata.refresh.interval.ms=500000;socket.timeout.ms=100000

Configuring the Topic-level property message.timeout.ms
 

Parameter	Value
rdkafkaTopicProperties	message.timeout.ms=400000