topic How to enable debug level testing endpoint connections? in Qlik Replicate

How to enable debug level testing endpoint connections?

danielrf — Mon, 29 Jul 2024 07:39:12 GMT

Hi,

I am moving Qlik Replicate form our current server to another linux which is a clean server. So now i need to reinstall and test all the drivers for databases and endpoints.

When i test my old working endpoint connections for Kafka as target, i get an error but all I see is this message:

Failed to get Kafka metadata: Local: Broker transport failure. _TRANSPORT

Which is very broad and unespecific,

So, How can i enable a more verbose error message testing enpoint connections?

thanks.

Re: How to enable debug level testing endpoint connections?

SachinB — Mon, 29 Jul 2024 08:22:24 GMT

Hello @danielrf ,

Thanks for contacting the Qlik community forum.

To enable debug level logs for testing Kafka endpoints, set the logging parameter to verbose. Then, click on "Test Connection" to initiate the test. After the test, click on "View Logs" in the same window to check the repsrv.log files for debug level messages

Hope this helps you.

Regards,

Sachin B

Re: How to enable debug level testing endpoint connections?

danielrf — Mon, 29 Jul 2024 10:58:51 GMT

Thanks!

Now i am able to see in more detail why the endpoinnt conection is not working....

Is weird, i have broker servers in place , but the log says I am not:

00420926: 2024-07-29T11:23:01:792443 [INFRASTRUCTURE ]T: Syntax object for 'KafkaSyntax' provider was not found in the repository (dbprops.c:424)

00420926: 2024-07-29T11:23:01:792466 [INFRASTRUCTURE ]T: The 'KafkaSyntax' provider syntax was not found in the repository. (dbprops.c:1160)

00420926: 2024-07-29T11:23:01:792473 [INFRASTRUCTURE ]T: The default 'KafkaSyntax' provider syntax will be used instead. (dbprops.c:1166)

00420926: 2024-07-29T11:23:01:807199 [SERVER ]I: Target endpoint 'Kafka' is using provider syntax 'KafkaSyntax' (provider_syntax_manager.c:955)

00420926: 2024-07-29T11:23:01:841995 [SERVER ]T: RDKAFKA - 5 - CONFWARN: [thrd:app]: No `bootstrap.servers` configured: client will not be able to connect to Kafka cluster (kafka_client.c:244)

00420926: 2024-07-29T11:23:11:843709 [SERVER ]T: Failed to get Kafka metadata: Local: Broker transport failure. _TRANSPORT [1020401] (kafka_client.c:437)

Re: How to enable debug level testing endpoint connections?

john_wang — Mon, 29 Jul 2024 11:21:34 GMT

Hello @danielrf ,

I'm not sure if this is a random error or if it has been persistent for a while. It seems to be more of a Kafka server or connectivity issue.

To troubleshoot, you might want to run the "ping" command from the Qlik Replicate server machine to check for any random connection issues. Alternatively, you could try replacing the hostname with the IP address in the Endpoint settings to see if that makes a difference.

In the past, we've experienced similar errors caused by a DNS server that wasn't working consistently.

Hope this helps.

Best regards,
John.

Re: How to enable debug level testing endpoint connections?

danielrf — Tue, 30 Jul 2024 08:43:26 GMT

Hi,

This behaviour is persistent 100% of the times. The thing is I am doing a migration from an old server where all the connectors and tasks are running for years. On the new server i installed the version 2023.11 to fix this issue: product issue (RECOB-8021)

Case # 00156465

Re: How to enable debug level testing endpoint connections?

john_wang — Tue, 30 Jul 2024 09:03:27 GMT

Hello @danielrf ,

Thank you so much for the sharing. It's very helpful for all of us.

I'm pasting the entire fix information:

Jira issue : RECOB-8021
Salesforce case : 108548
Type : Issue
Component/Process: Kafka
Description : Tasks using the Kafka target endpoint would consume excessive memory when capturing the REAL data type and using JSON message format.

Regards,

John.

Re: How to enable debug level testing endpoint connections?

danielrf — Tue, 30 Jul 2024 14:34:11 GMT

Hi , now i did a test using a kafka cli client and using the same configuration as Qlik Replicate enpoint, from the console i am able to list topics but not working from Replicate.

]$ ./kafka-topics.sh --list --bootstrap-server broker-0-kafka-east-sit:9093 --command-config ssl.properties | more
cdc.fin.jdetest.attrep_apply_exceptions
cdc.fin.jdetest.crpctl.f0004
cdc.fin.jdetest.crpctl.f0005

Re: How to enable debug level testing endpoint connections?

john_wang — Thu, 01 Aug 2024 10:11:32 GMT

Hello @danielrf ,

Did you run the Kafka client program from Replicate Server machine?

thanks,

John.

Re: How to enable debug level testing endpoint connections?

danielrf — Thu, 01 Aug 2024 10:14:41 GMT

Hi ,

Yes the test was done from the same new server. And i was able to connect and list the topics with the console kafka client with the same certificates as from qlik replicate.

Re: How to enable debug level testing endpoint connections?

john_wang — Thu, 01 Aug 2024 10:36:46 GMT

Hello @danielrf ,

Thank you for the clarification. Would you please open a support ticket with below information:

1- The Replicate task Diag Package: we need to check your Kafka endpoint configuration; And also the "repsrv.log" which will record the endpoint test failure information there.

2- The information you used in the Kafka client: we need to compare to Replicate endpoint.

Our support team will be more than happy to assist you.

Regards,

John.

Re: How to enable debug level testing endpoint connections?

danielrf — Wed, 07 Aug 2024 14:48:09 GMT

Hi, I opened a case, but it seems the support team is slower than these forums.

About the issue: i compared the manual connection and i see i have this parameter

ssl.endpoint.identification.algorithm=

(space blank , null as value)

while on the qlik replicate new version i tried to inject via rdkafkaProperties like this

ssl.endpoint.identification.algorithm=https

or null but no success.

Is there another way to inject this parameter?

Re: How to enable debug level testing endpoint connections?

john_wang — Wed, 07 Aug 2024 22:50:06 GMT

Hello @danielrf ,

In Kafka client properties, ssl.endpoint.identification.algorithm= is a Global configuration property. so it can be injected by internal parameter rdkafkaProperties.

And it is a parameter in Kafka server settings too, see similar discussion in What are the default SSL configurations for a Kafka target. Change its setting requires the Kafka brokers restarting.

You may try to put it in Server side to see if it makes difference.

Hope this helps.

John.

Re: How to enable debug level testing endpoint connections?

danielrf — Thu, 08 Aug 2024 09:55:04 GMT

Hi ,

setting on rdkafkaProperties

ssl.endpoint.identification.algorithm=none

worked for me

Thanks!

Re: How to enable debug level testing endpoint connections?

john_wang — Thu, 08 Aug 2024 09:56:59 GMT

Thank you so much for your clarification @danielrf !