Replicate parent streaming task failed. Unable to start due to memory error.

[desmondchew]

We have 256GB of RAM in the Windows Server 2019. All our tasks are hovering max 50% of utilization of memory. Unfortunately we encountered parent streaming task failure due to memory. I tried to resume with CDC it failed to start.

Not sure how to resolve.  Looks like need a case open.

00010576: 2023-08-28T20:44:46 [SOURCE_CAPTURE ]I: Scale is set to 10 for NUMBER Datatype (oracle_endpoint_imp.c:1307)
00010576: 2023-08-28T20:44:46 [SOURCE_CAPTURE ]I: Retry interval is set to 5 (oracle_endpoint_imp.c:1321)
00010576: 2023-08-28T20:44:46 [SOURCE_CAPTURE ]I: Oracle source database version is 11.2.0.4.0 (oracle_endpoint_conn.c:611)
00010576: 2023-08-28T20:44:46 [SOURCE_CAPTURE ]I: Oracle Client version: 19.10.0.0.0 (oracle_endpoint_conn.c:624)
00010576: 2023-08-28T20:44:49 [SOURCE_CAPTURE ]I: Oracle compatibility version is 11.2.0.4.0 (oracle_endpoint_conn.c:87)
00010576: 2023-08-28T20:44:49 [SOURCE_CAPTURE ]I: Database role is 'PRIMARY' (oracle_endpoint_conn.c:133)
00010576: 2023-08-28T20:44:49 [SOURCE_CAPTURE ]I: Resetlog process is supported in Oracle Database (oracle_endpoint_imp.c:1503)
00015260: 2023-08-28T20:44:49 [STREAM_COMPONEN ]I: Target last committed record id from the previous run is '8528033961' (streamcomponent.c:1689)
00015260: 2023-08-28T20:44:49 [TARGET_APPLY ]I: Timeline is 20230816232738509140 (ar_cdc_channel.c:505)
00015260: 2023-08-28T20:44:49 [TARGET_APPLY ]I: Working in transactional apply mode (endpointshell.c:2032)
00015260: 2023-08-28T20:44:49 [SOURCE_CAPTURE ]I: Source endpoint 'Oracle' is using provider syntax 'Oracle' (provider_syntax_manager.c:951)
00010160: 2023-08-28T20:44:49 [SORTER ]I: Start the task using saved state. Start source from stream position 0000001c.02da86e0.00000001.0078.02.0000:846080.321354.16 and id 2315089047. Confirmed target id is 8528033961, next target id is 8528033965, last source timestamp 1693262921000000 (sorter.c:532)
00010160: 2023-08-28T20:44:49 [INFRASTRUCTURE ]E: Not enough memory resources are available to process this command. (apr status = 720008) [1000104] (at_memory.c:428)
00007084: 2023-08-28T20:44:49 [TASK_MANAGER ]I: Task error notification received from subtask 0, thread 2, status 1020100 (replicationtask.c:3520)
00010160: 2023-08-28T20:44:49 [SORTER_STORAGE ]E: Cannot allocate transaction storage ReadBuffer [1020100] (transaction_storage.c:6076)
00010160: 2023-08-28T20:44:49 [SORTER_STORAGE ]E: The Sorter Transaction Storage cannot read events from swap. File name D:\Attunity\Replicate\data\tasks\ODS_ALL/sorter/ars_swap_tr_00000000000000003379.tswp, Transaction Id 80.321354.16 and id 2315089047. Confirmed target id is 8528033961, next target id is 8528033965, last source timD:\Attunity\Replicate\data\tasks\ODS_ALL/sorter/ars_swap_tr_00000000000000003379.tswp [1020100] (transaction_storage.c:6502)
00010160: 2023-08-28T20:44:49 [SORTER_STORAGE ]E: Cannot restore swap file 'ars_swap_tr_00000000000000003379.tswp [1020100] (transaction_storage.c:6202)
00010160: 2023-08-28T20:44:49 [SORTER ]E: The Transaction sorter cannot initialize Transaction storage [1020100] (sorter.c:557)
00010160: 2023-08-28T20:44:49 [TASK_MANAGER ]E: Failed while preparing stream component 'st_0_transaction_sorter'. [1020100] (subtask.c:890)
00010160: 2023-08-28T20:44:49 [TASK_MANAGER ]E: Cannot initialize subtask [1020100] (subtask.c:1334)
00010160: 2023-08-28T20:44:49 [SORTER ]E: Stream component 'st_0_transaction_sorter' terminated [1020100] (subtask.c:1602)
00007084: 2023-08-28T20:44:49 [TASK_MANAGER ]I: Subtask #0 ended (replicationtask_util.c:591)

Desmond

[john_wang]

Hello @desmondchew ,

Thanks for opening the article.

I'm not sure how much memory was available when this task was stopped. Please take note that if the changes were offloaded to harddisk and the task is resuming again, Replicate will read all the TSWP file into memory. If the TSWP file is pretty big then we may face "Not enough memory resources are available to process this command" & "[SORTER_STORAGE ]E: Cannot restore swap file 'ars_swap_tr_00000000000000003379.tswp".

In this case I'd like suggest to restart the task from given SCN (you can get the SCN from the task log) to let Replicate recapture the changes (and discard the .tswp file), or RELOAD the task (if the redo log is not available already).

Hope this helps.

Regards,

John.