ODBC Time-out and no error notification?

[RonFusionHSLLC]

Strange issue today...

Got an ODBC Timeout, no email, no notification on the UI...only evidence was users complaining and this in the log files.

Log levels are right down the middle across the board...why didn't Replicate report the error and are there time-out setting(s) to allow this to be a bit more resilient?  We had a db-log backup that started up right about this time, which by itself is normal...but might have used enough resources to cause Replicate operations to time-out...only theory we have at the moment.

00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: RetCode: SQL_ERROR SqlState: HYT00 NativeError: 0 Message: [Microsoft][ODBC Driver 17 for SQL Server]Query timeout expired [1022502] (ar_odbc_stmt.c:2828)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: Failed to execute bulk insert statement.
~--removed encrypted statment ~ [1022506] (odbc_util.c:4504)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: Failed to apply INSERT (1) for table 1 [1022506] (bulk_apply.c:2997)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: Failed to start bulk apply transaction [1022502] (sqlserver_bulk.c:396)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: Rollback bulk statements failed [1022502] (bulk_apply.c:3074)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: Error forwarding bulk [1022506] (bulk_apply.c:815)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: Endpoint is disconnected [1020414] (endpointshell.c:4113)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: Error executing data handler [1020414] (streamcomponent.c:2026)
00011328: 2023-12-06T01:12:33 [TASK_MANAGER ]I: Stream component failed at subtask 0, component st_0_test_to_phygenesys_repl [1020414] (subtask.c:1433)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]I: Target component st_0_test_to_phygenesys_repl was detached because of recoverable error. Will try to reattach (subtask.c:1548)
00011328: 2023-12-06T01:12:33 [TARGET_APPLY ]E: Failed to get table (null).attrep_changes93EDA050474F5777 [1022502] (ar_odbc_conn.c:1060)
00011328: 2023-12-06T01:12:38 [STREAM_COMPONEN ]I: Target last committed record id from the previous run is '26829843' (streamcomponent.c:1689)
00011328: 2023-12-06T01:12:38 [TARGET_APPLY ]I: Bulk is set to ignore max row size warnings (sqlserver_endpoint_imp.c:1752)
00011328: 2023-12-06T01:12:38 [TARGET_APPLY ]I: Working with collated columns (sqlserver_endpoint_imp.c:1812)
00011328: 2023-12-06T01:12:38 [TARGET_APPLY ]I: Working with BCPKeepNull set to TRUE (sqlserver_endpoint_imp.c:1816)
00011328: 2023-12-06T01:12:38 [TARGET_APPLY ]I: Target endpoint 'SQL Server' is using provider syntax 'SQLServer' (provider_syntax_manager.c:957)
00011328: 2023-12-06T01:12:38 [TARGET_APPLY ]I: Going to use provider syntax for datatype mapping (sqlserver_endpoint_imp.c:1843)

[Dana_Baldwin]

Hi @RonFusionHSLLC 

You should have received a notification if you've got them configured, please confirm: Notifications settings #Notifications settings | Qlik Replicate Help

If you've got notifications configured, and you didn't get one, please open a support case. If it's a software defect we'll need to involve our internal support team and there's no pathway for that in our Community forum.

What type of target endpoint are you using? Several of our endpoints have timeout settings but they vary. For example, if the target endpoint is ODBC there are the following internal parameters you can set & increase from the defaults on the advanced tab:

executeTimeout, default 60 seconds

loadTimeout, default 1200 seconds

Type an exclamation point into the search parameter field and you can scroll down to them.

The new settings will not take effect until the tasks that use them are stopped and resumed (endpoints are only read/processed during task startup).

Hope this helps.

Dana

[RonFusionHSLLC]

Here's what I have setup for notifications, I typically get e-mails and indication on the UI.

Both source/target are MS SqlServer 2022 (with ms-cdc) over MS ODBC driver

Thanks for the heads up on the params.

Ron

[Dana_Baldwin]

Hi @RonFusionHSLLC 

Please open a support case so we can look into the issue further.

For SQL Server target endpoint, there is also an internal parameter

cdcTimeout, default 600 seconds

Thanks,

Dana

[Heinvandenheuvel]

Well, it's deemed (rightly or wrongly so) a recoverable error. I don't expect an error message from that. On the UI you may have briefly had a red (X) mark next to the endpoint.

IMHO it is best to catch and report on these kind of issues through LATENCY alerts.

I would agree with you that is critical to know that this is happening and grepping through a log is not good way for that.

I do think it is good operational practice to daily go over every single log file produced and look for ]E: and even ]W: lines. Such script may learn to ignore some 'heavy hitters'or just count them, not show all.

Hth

Hein.