topic Re: Latency in child Task in Qlik Replicate

Latency in child Task

Titus — Sat, 06 Jan 2024 21:49:21 GMT

Hi Team,
The child task has 100k records in the on-disk target. Despite committing in the last 6 hours, no records were applied to the target. The task has automatically switched to 'one by one' mode. I have attached a log; please review it and let me know why records are going to on-disk. I attached only suspected part in the log.

Thanks,
Titus

Re: Latency in child Task

john_wang — Sun, 07 Jan 2024 01:46:57 GMT

Hello Titus @Titus ,

Thanks for reaching out to Qlik Community! It's a great idea to upload a partial task log file so far the sensitive information (database IP Address, user name etc) was removed however some basic information was deleted too, we do not know the source and target sides database type/version etc.

From the limited information, looks to me the latency was led by applying failure. In general Qlik Replicate will apply changes in Batch Apply Mode, if hitting errors, the apply mode will switch to 'one-by-one' mode automatically and Replicate will handle every row of the Batch, until each row is applied successfully or is recorded into attrep_apply_exceptions table (if 'one-by-one' failed still), then switch back to Batch Apply Mode automatically. If there is a huge volume changes waiting for apply and hitting this error repeatedly, the switching must lead big latency:

1- the batch apply failed, then Replicate switch to 'one-by-one' mode automatically

2024-01-06T14:16:55 [TARGET_APPLY ]I: Bulk apply operation failed. Trying to execute bulk statements in 'one-by-one' mode

2- BTW, the 'one-by-one' apply SQL is pretty long:

2024-01-06T14:26:14 [TARGET_APPLY ]T: RetCode: SQL_ERROR SqlState: 22000 NativeError: 100078 Message: String 'UPDATE "SAP_AP"."LIPS" SET "MANDT"='100', "VBELN"='8023490890', "POSNR"='000020', "PSTYV"='ZTBP', "ERNAM"='PREETHJ', "ERZET"='005208', "ERDAT"='20240106', "MATNR"='000000000000141002', "MATWA"='00000...' is too long and would be truncated [1022502] (ar_odbc_stmt.c:4660)

Not sure what's the source table structure, maybe it's a very wide table (eg many columns in Oracle DB)? however I cannot see the original applying SQL as it's encrypted:

~{AQAAANatIQ1mn9q6pGgUSgveMJi34akhSVomjc+nixRxuvIgwGgyM4ZuM1qtyAv5frXWJ7XMWacb2u7...

You may run repctl dumplog to decrypt the task log file and check the SQL to understand the RCA further.

if you still need help, please open a support ticket with below information:

Source table creation DDL
Set SOURCE_CAPTURE/TARGET_APPLY to Verbose, recreate the error and then attach the Diag Packages to the support ticket
Attach the decrypted task log file (which was generated in above step)

Hope this helps.

John.

Re: Latency in child Task

SushilKumar — Sun, 07 Jan 2024 05:25:41 GMT

Hello Team,

Thanks for reaching out to Qlik via Qlik Community Support page. Such issue require through analysis of Task Settings, hence request you to reach out to technical Support via a case and provide Diagnostics package .

Please refer to Replicate https://community.qlik.com/t5/Knowledge/How-to-collect-Diagnostics-Package-from-Qlik-Replicate/ta-p/1815056

Regards,

Sushil Kumar

Re: Latency in child Task

deepaksahirwar — Mon, 08 Jan 2024 03:06:10 GMT

Hi @Titus ,

For the Qlik Data Integration child task is experiencing latency issues.
This could be due to a variety of factors. Here are some potential causes and solutions

Task Mode: If the task is running in ‘one by one’ mode, it could cause latency. This mode is usually triggered when batches break.

Source or Handling Latency: Check if the latency is on the source or handling side. If it’s handling latency, consider increasing buffers and buffer size. If it’s source latency, check for any current outages/service interruptions or if source maintenance is being carried out.

Connection Issues: There might be connection issues to the target database. This could be due to DNS issues or other network-related problems.

Volume of Changes: The volume of changes and the speed of the target database can affect performance.

Logs and Diagnostics: Set SOURCE_CAPTURE to Verbose and resume the task, then upload the Diagnostic Packages. Also, upload the previous run task log file.

Hope this helps.

Deepak

Help users find answers! Do not forget to mark a solution that worked for you! If already marked, give it a thumbs up!