topic Re: Retry Pushing Data To Kafka for spesific Errors in Qlik Replicate

Retry Pushing Data To Kafka for spesific Errors

gseckin — Tue, 08 Oct 2024 09:17:46 GMT

Hi,

We are using Kafka as a target and sometimes Kafka throws timeout errors and policy violation errors (when Kafka cannot handle the load), when this happens Qlik failed to deliver the message. Is there a way to retry sending the message to Kafka when Qlik receives an error from Kafka?

Re: Retry Pushing Data To Kafka for spesific Errors

john_wang — Tue, 08 Oct 2024 09:38:58 GMT

Hello @gseckin ,

Thanks for reaching out to Qlik Community!

Would you please try to add an internal parameter rdkafkaTopicProperties to the Kafka target endpoint, and set its values to:

request.timeout.ms=240000;message.timeout.ms=1200000;

Hope this helps.

John.

Re: Retry Pushing Data To Kafka for spesific Errors

gseckin — Tue, 08 Oct 2024 09:43:11 GMT

Thanks I'll try and can we also define kafka max throughput espacially for full loads kafka can not handle the load so we need to decrease max throughput for kafka

Re: Retry Pushing Data To Kafka for spesific Errors

gseckin — Tue, 08 Oct 2024 11:57:22 GMT

Hi @john_wang,

is there also retry mechanism for kafka that we can use for spesific errors?

Re: Retry Pushing Data To Kafka for spesific Errors

john_wang — Thu, 10 Oct 2024 04:45:29 GMT

Hi @gseckin ,

In the error handling, the retry can be customized, for example:

Hope this helps.

John.

Re: Retry Pushing Data To Kafka for spesific Errors

gseckin — Thu, 10 Oct 2024 10:02:08 GMT

Hi @john_wang ,

thanks! And what about setting max throughput for Kafka? Can we control the max amount of data per min or sec that qlik pushes to Kafka?

Thanks

Re: Retry Pushing Data To Kafka for spesific Errors

john_wang — Thu, 10 Oct 2024 13:53:46 GMT

Hello @gseckin ,

No such setting in Qlik Replicate Kafka endpoint. However there are other options out of Qlik Replicate:

1. Producer-Side Configurations:

linger.ms: This controls the time the producer waits for more records before sending a batch. Increasing it can reduce the throughput but increase batching efficiency.
batch.size: This controls the maximum size of a batch that can be sent. Reducing the batch size limits the throughput.

2. Broker-Side Configurations:

quota.producer.default: Kafka has a built-in quota mechanism for throttling producers. This allows you to set throughput limits on a per-client or per-IP basis.

3. Throttling Network Bandwidth:

If the network between your producers and Kafka broker is the bottleneck, you can throttle network bandwidth at the OS level using tools like tc on Linux.

By combining producer throttling, broker quotas, and potentially network-level restrictions, you can efficiently limit Kafka's throughput to meet your requirements.

Hope this helps.

John.