Qlik replicate throughput limitation at around 7000kbyte/sec

[WLYB]

Dear Qlik Support,

Thank you for assisting with the Qlik Replicate issue.

We are experiencing  source and target throughput limitation to an average of approximately 7,000 kbyte/s,  time  to time  we can reach the pick of  12000 kbyte/sec. We have investigated but have not been able to identify the root cause.

System resources checked

• Application server: 8-core CPUs, 35 GB RAM, 500 GB disk space
• Load balancer server: 2-core CPUs, 3.5 GB RAM No obvious issues were found on either side.

Current data flow

• From Oracle (ODS) to local disk: T_LSS_ODS
• From local disk: S_ODS_LSS to Azure Event Hub Note: Both T_LSS_ODS and S_ODS_LSS reside on the same Qlik application server.

Observation The same throughput limitation occurs for both replication paths: ODS → T-LSS_ODS and S_ODS_LSS → Azure Event Hub ( 20 tasks)  are  running in Batch optimized  .

This suggests the issue may be located at the Qlik Replication server itself. Impact The throughput issue affects a task that replicates a large number of records daily. ( +/- 5 Millions of records ) .

Qlik Replication configuration

• messageKey: SCHEMA_AND_TABLE
• partitionMapping: MESSAGE_KEY
• rdkafkaProperties: ssl.endpoint.identification.algorithm=none; queue.buffering.max.messages=500000; acks=1; linger.ms=100
• sslCAPath: /app/attadm/attunity/replicate/ca-azure.pem
• topic: wlbeodsqlik
• useSSL: true

NB:   There is not message in the Qlik log file related to PERFORMANCE  message  .  what does means no performance problem !!

We would really appreciate your assistance in diagnosing and resolving this throughput issue.

Kind regards,

Youssef

[Dana_Baldwin]

Hi @WLYB 

Thanks for posting in Qlik Community. For some endpoints, there are things you can try after you know if it is source or target latency. For some performance issues and endpoints, specific tuning or task design considerations might be needed, which is handled by our Professional Services team as they can be use case specific, requiring a deeper analysis. Please refer to these links for what Support & Professional Services cover:

How and when to contact Qlik's Professional Servic... - Qlik Community - 1714936

https://community.qlik.com/t5/Official-Support-Articles/How-to-contact-Qlik-Support/ta-p/1837529

The performance logging component does not write any information to the task logs when set to the default level of "info". If you increase this to trace, you will see if the latency is on the source, the target, or on the Replicate server itself. When source latency=target latency, all the latency is at the source (target latency can never be lower than source latency).

This will also include information you can use to determine the read speed from the Oracle source. We consider "good" read speed from Oracle as 100 MB per second:

[PERFORMANCE     ]T:  Completed to read from archived Redo log 51200000 bytes at offset 000000009eb09c00 with rc 1, read time is 1109 ms, thread '1'

Convert bytes to megabytes, 51.2, then convert milliseconds to seconds, 1.109, and divide. In this example that is 46.1 MB per second, which is less than desirable.

For things to try for source latency reading from Oracle:

Latency / Performance Troubleshooting and Tuning f... - Qlik Community - 1734097

One practical way to identify slow network issues is to take a relatively large file, 2GB for example (so it doesn't copy so fast you can't measure the rate) and measure the amount of time it takes to copy it from the source (or target) database server to the Replicate server. Present this to your network team.

I hope this helps! Please consider working with our Professional Services team if a deeper dive is needed.

Thanks,

Dana

[john_wang]

Hello @WLYB ,

I agree with @Dana_Baldwin . In addition to Dana’s comments:

1. Please provide the network bandwidth between Qlik Replicate and the source database, as well as between Qlik Replicate and the target Azure Event Hub to PS team.

2. Please also note the “Throughput Units” configured for the target endpoint, as this setting directly impacts overall performance. The default value is 1.

Hope this helps.

John.

[WLYB]

Thanks for your reply. In my Qlik version (May 2024), the Throughput units do not exist in the Qlik Management Console.

Attached is the network bandwidth between Qlik Replicate and the source/target databases.

The performance issue is mainly located between Qlik and the target Azure EHUB.

Kind regards,
Youssef

   

[john_wang]

Hello @WLYB ,

I'd like to suggest upgrading Replicate to higher version to utilize the new feature.

Regards,

John.