Upgrading Kafka library on QLIK Replicate

[cloudera-admin]

We have some problems with the version of the Kafka library used by QLIK (v1.5.2).

I would like to know if it is possible to upgrade to version 1.9.0 ?

And if yes, how ?

[john_wang]

Hello @cloudera-admin ,

Qlik Replicate utilizes librdkafka APIs to communicate with Kafka clusters(or similar streaming systems) however I'm afraid there is no way to replace the embedded client libraries.

In current Replicate versions, 2022.5 & 2022.11 , the Kafka client version is 1.5.2.

BTW, looks to me the most new Kafka client librdkafka version is 1.6, seems no version 1.9.0.

I'd like to suggest opening a support case so far we may get futher investigations on the issues without replace the libraries.

Regards,

John.

[SwathiPulagam]

Hi @cloudera-admin ,

Many customers are using librdkafka version 1.5.2 and please explain what is blocking you to use 1.5.2.

We want to know more details about the issue you are facing.

Thanks,

Swathi

[cloudera-admin]

Hello @john_wang 

As far as I understand github, it seems that 1.9.2 was available at the time of your writing : Releases · confluentinc/librdkafka · GitHub

@SwathiPulagam : We are facing Kafka timeouts, same issue as Reached Time out while waiting for acks from kafka - Qlik Community - 1889941. So we increased resultsWaitTimeoutMS (360000ms in our case), but we now encounter longer timeouts. We are working with our network team and Confluent to see what is leading to this situation. But we want to tackle this with any means because it now have huge impacts on our production. So kafka library update is still an option.

So, is there a known and supported process to upgrade this library on Qlik Replicate (github show v2.0.2 as the latest now, but our target is at least 1.9.x) ?

Regards

[john_wang · Accepted solution]

Hello @cloudera-admin , 

Thanks for your update.

First of all my apologize, you are right, there are v1.9.x and v2.0.2 in "Tags" tree in page librdkafka .

We can replace Kafka client libraries by v1.9.x and/or v2.0.2 in our support test labs however it's unsupported process, we should not expose it here. Instead the recommended options are:

1. Open support ticket and our support team will help you solve the timeout issue without Kafka client library replacement;

2. Even if support team verify it's a Kafka client issue then we need PM team review and approval to add higher version client official support.

In short, replacing Kafka client is risk as it does not pass Qlik QA regression test it should not be used.

Regards,

John.