Failed to drop external file format error and no events are captured but task is in running state

[Kohila]

Hi Team,

We are having a replicate task with Source as SAP application and target as Synapse with LS and LR task.  Our task have recovered with retry # from failure below, after we could see inconsistency in source and target table record counts

00002648: 2023-05-28T01:14:20 [TARGET_APPLY ]I: RetCode: SQL_ERROR SqlState: HYT00 NativeError: 0 Message: [Microsoft][ODBC Driver 17 for SQL Server]Query timeout expired

00002648: 2023-05-28T01:44:48 [TARGET_APPLY ]W: Failed to drop external file format ATTREP_FF_G2_edw_7a20c2b2_8fda_bc4e_af48_1e545a0183eb (cloud_imp.c:5459)
00002648: 2023-05-28T01:44:48 [TARGET_APPLY ]W: Failed to drop external file format

Our question is, why the transaction is not applied in target after the failure, only few records are missed to apply. Qlik will not catch up to apply the transaction to target where it stopped. Any insight on behind the scene causing this issue?

[sureshkumar]

Hello Team,

00002648: 2023-05-28T01:14:20 [TARGET_APPLY ]I: RetCode: SQL_ERROR SqlState: HYT00 NativeError: 0 Message: [Microsoft][ODBC Driver 17 for SQL Server]Query timeout expired

Please set the below internal parameter at target endpoint and see if this resolves the issue?

We are suspecting this is because of target is too busy, Could you please do have any long-running jobs are going on at target side?

1. Stop the task.

2. open target endpoint, / Advanced tab / internal Parameters and add the below:

cdcTimeout (default 600); 6000

executeTimeout (default 60); 2400

loadTimeout (default 1200) : 12000

3. try to resume task and see if we still have a timeout issue.

Regards,

Suresh

[Kohila]

Any reason why we have missing inserts in target ? 

[john_wang · Accepted solution]

Hello @Kohila ,

It's hard to tell by error messages this time. We'd like to suggest you open a support ticket with below information (we may need additional information based on support team analysis):

1- In Target Endpoint set both "keepCSVFiles" and "keepErrorFiles" to true

2- Set TARGET_APPLY / SOURCE_CAPTURE to Verbose and recreate the issue, download and attach Diag Packages to the support case

3- Attach the source table and target table creation DDL (include PK, Unique index etc).

      It's better to let us know which line (eg the PK value, or a given keyword) was missed

3- Attach the generated CSV files as well, they are very important to understand in which stage the data missed.

Regards,

John.

[Kohila]

Thanks John, We have raised Qlik case and team is working on it.