An existing tasks failed to start out of a sudden

[desmondchew]

We have an existing that that failed to start out an a sudden. It was able to run previously. Then the source DB encountered max_connections issues. After we have fixed that, this task is unable to start with full load.

However we have other tasks that connects to the same source and target endpoints are working though.

Source and target DB endpoints are same - MySQL AWS 8.0, ODBC driver 8.0.23

Here's the error log.

Error 1236 (Could not find first log file name in binary log index file) reading binlog

0007120: 2023-06-05T21:14:14 [SOURCE_CAPTURE ]I: System variable 'binlog_checksum' = 'NONE' (mysql_endpoint_capture.c:382)
00007120: 2023-06-05T21:14:14 [SOURCE_CAPTURE ]I: Error 1236 (Could not find first log file name in binary log index file) reading the binlog, try to reconnect (mysql_endpoint_capture.c:1150)
00007120: 2023-06-05T21:14:19 [SOURCE_CAPTURE ]E: Error 1236 (Could not find first log file name in binary log index file) reading binlog [1020403] (mysql_endpoint_capture.c:1160)
00009432: 2023-06-05T21:14:19 [TASK_MANAGER ]I: Task error notification received from subtask 0, thread 0, status 20014 (replicationtask.c:3517)
00007120: 2023-06-05T21:14:19 [SOURCE_CAPTURE ]E: Error executing source loop [20014] (streamcomponent.c:1940)

Please help.

Latest update: Now all the tasks connecting to the same source is unable to start. I am suspecting some issue with the source endpoint AWS. We have the error.

"The gp2 burst balance credits for the RDS database instance are low. To resolve this issue, reduce IOPS usage or modify your storage settings to enable higher performance."

Desmond

[SachinB]

Hello @desmondchew ,

Hope you are doing good, We do have a couple of information required in order to trouble shoot.

Error 1236 in MySQL typically occurs when the replication process is unable to locate the first log file name in the binary log index file. This error is commonly associated with MySQL's replication feature

The tasks which are connected and working properly, is it on same replicate version? 
What is your current replicate version?
Could you please set source capture in verbose and provide enhanced logs?

It will be better if create support ticket and will work on this.

Regards,

Sachin B

[john_wang]

Hello @desmondchew ,

Thanks for opening the article.

Not very sure if you are running some special builds (please check the MySQL source defect and fix (2022.5 & 2022.11) ), if yes, please upgrade Replicate; if NOT then seems the binlog was not available anymore.

You may see a line in Replicate tas log file which contains the binlog file number, like:

2023-06-05T13:46:01:868602 [SOURCE_CAPTURE  ]I:  Stream positioning at context '$.000034:3506:-1:3506:0'

where 000034 is the file suffix in MySQL binlog files name.

You may check if the file available by command:

show master status;

Hope this helps.

Regards,

John.

[desmondchew]

We are running on QDI November 2022 (2022.11.0.394).

Both source and target are RDS MySQL which doesn't allow us to retrieve the binlog status.

The tasks was running fine even with existing QDI version November 2022 (2022.11.0.394). Then it stopped running out of a sudden.

[DesmondWOO]

Hi @desmondchew ,

Could you please provide further explanation as to why retrieving the binlog status is not possible on your RDS MySQL? Is it related to a permission issue? 

As stated in the Amazon RDS user guide, running SHOW MASTER STATUS should be possible. 
https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-stored-proc-replicating.html

I suggest enabling VERBOSE logging on the SOURCE_CAPTURE logger to get more information for the error message.

Regards,
Desmond

[desmondchew]

Hi,

There's no indication that the guide allows 'SHOW MASTER STATUS' in a RDS AWS. https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/mysql-stored-proc-replicating.html

I however run select * from mysql.rds_history. I got null value for the master_host columns.

[desmondchew]

Hi,

Apparently there's an issue with only one table. I am attaching the logs here in tracing mode. Is the source table corrupted?

00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: Connection failure error codes are: 1317,1053 (provider_syntax_manager.c:2902)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: Data failure error codes are: 1441,1292,1406,1264,1265 (provider_syntax_manager.c:2932)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: Data failure error codes are: 1441,1292,1406,1264,1265 (provider_syntax_manager.c:2932)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: ODBC source. Cannot init table workspace - '1022511' - for table 1 [1022511] (mysql_endpoint_unload.c:306)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]E: RetCode: SQL_ERROR SqlState: 3D000 NativeError: 1046 Message: [MySQL][ODBC 8.0(w) Driver][mysqld-8.0.32]No database selected [1022502] (ar_odbc_stmt.c:4011)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]E: ODBC source. Cannot init table workspace - '1022511' - for table 1 [1022511] (mysql_endpoint_unload.c:306)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: Error executing source loop [1022511] (streamcomponent.c:1940)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: Stream component 'st_1_ods-replica-prod' terminated [1022511] (subtask.c:1565)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: Entering 'mysql_endpoint_free_imp' (mysql_endpoint_imp.c:785)
00002140: 2023-06-06T10:23:21 [TASK_MANAGER ]I: Task error notification received from subtask 1, thread 0, status 1022511 (replicationtask.c:3517)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: Leaving 'mysql_endpoint_free_imp' (mysql_endpoint_imp.c:802)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]E: Error executing source loop [1022511] (streamcomponent.c:1940)
00007304: 2023-06-06T10:23:21 [TASK_MANAGER ]E: Stream component failed at subtask 1, component st_1_ods-replica-prod [1022511] (subtask.c:1396)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]E: Stream component 'st_1_ods-replica-prod' terminated [1022511] (subtask.c:1565)
00007280: 2023-06-06T10:23:23 [SOURCE_CAPTURE ]T: Last binlog = 'mysql-bin-changelog.236160', len = 6030016 (mysql_endpoint_capture.c:725)
00007280: 2023-06-06T10:23:23 [SOURCE_CAPTURE ]T: Wait 5 seconds in idle time (mysql_endpoint_capture.c:1048)

[Steve_Nguyen]

the error :

00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]T: ODBC source. Cannot init table workspace - '1022511' - for table 1 [1022511] (mysql_endpoint_unload.c:306)
00007304: 2023-06-06T10:23:21 [SOURCE_UNLOAD ]E: RetCode: SQL_ERROR SqlState: 3D000 NativeError: 1046 Message: [MySQL][ODBC 8.0(w) Driver][mysqld-8.0.32]No database selected [1022502] (ar_odbc_stmt.c:4011)

---

if only on one table, try to create at test task for this one table.

- set source_capture , and source_unload ,, verbose logging to see why it error on this table.