How to resolve error 1000123 ( undefined symbol: db2ReadLog)?

[parthiTA]

Can you please help us understand why we see erro 100123 (undefined symbol: db2ReadLog?

Trying to replicate from on-prem DB2-LUW to DB2-LUW on cloud, using user credentials that has read access on the source database, and full permission on the target database.

Is this error related to user privileges or the client driver? If related to user privileges, what additional privileges are needed?

Appreciate your help.

00008036: 2022-03-03T12:47:54 [TASK_MANAGER ]I: All stream components were initialized (replicationtask.c:3609)
00008039: 2022-03-03T12:47:54 [AT_GLOBAL ]I: Dynamic loading 'libdb2o.so' (db2luw_endpoint_util.c:1949)
00008041: 2022-03-03T12:47:54 [SORTER ]I: Sorter last run state: confirmed_record_id = 0, confirmed_stream_position = '' (sorter_transaction.c:3251)
00008036: 2022-03-03T12:47:54 [TASK_MANAGER ]I: Task error notification received from subtask 0, thread 0, status 1000123 (replicationtask.c:3433)
00008036: 2022-03-03T12:47:54 [TASK_MANAGER ]W: Task 'DDOLT onprem - DDOLD AWS' encountered a recoverable error (repository.c:5795)
00008041: 2022-03-03T12:47:54 [SORTER ]I: Final saved task state. Stream position , Source id 0, next Target id 1, confirmed Target id 0, last source timestamp 0 (sorter.c:770)
00008039: 2022-03-03T12:47:54 [INFRASTRUCTURE ]E: Failed to get symbol <db2ReadLog>: /apps/db2/odbc_cli/clidriver/lib/libdb2o.so: undefined symbol: db2ReadLog [1000124] (at_loader.c:223)
00008039: 2022-03-03T12:47:54 [INFRASTRUCTURE ]E: <Failed to get symbol <db2ReadLog>: No such file or directory> and the dll name: <libdb2o.so> [1000123] (at_loader.c:484)
00008039: 2022-03-03T12:47:54 [AT_GLOBAL ]E: Error Initializing endpoint. Failed to load libdb2o.so [1000123] (db2luw_endpoint_util.c:1954)
00008039: 2022-03-03T12:47:54 [SOURCE_CAPTURE ]E: Error loading DB2 CLI library [1000123] (db2luw_endpoint_capture.c:2709)
00008039: 2022-03-03T12:47:54 [SOURCE_CAPTURE ]E: capture_start: in agent_setup [1000123] (db2luw_endpoint_capture.c:3064)
00008039: 2022-03-03T12:47:54 [SOURCE_CAPTURE ]E: Error executing command [1000123] (streamcomponent.c:2024)

[Heinvandenheuvel]

Did it ever work, or is this a first try?

Does the 'test' on the (source?) end-point work? Any failure detail will be in <data>\logs\repsrv.log

It feels like an install/prerequisite problem, an incorrect or missing PATH / LD_LIBRARY_PATH

Driver installed?

- For IBM DB2 for LUW 10.5, install client version 10.5
- For IBM DB2 for LUW 11.1 and 11.5, install client version 11.1

Replicate Server restarted after driver install to pick up any new PATH setting?

Can you locate - libdb2o.so and does the replicate server account have permission to read/execute it?

Hein.

[john_wang]

Hello @parthiTA , copy @Heinvandenheuvel ,

Besides Hein comments, I'd like to believe there is another possibility: you are running a wrong ODBC Client, Please do not use ODBC CLI (it's not Replicate prerequisites) , we need IBM Data Server Client. See similar issue undefined symbol: db2ReadLog .

Hope this helps,

Regards,

John.

[parthiTA]

John and Hein, Thanks for your responses.

No, this is the first time we are setting up the replication from DB2 LUW source to DB2 LUW target.

Based on the input, we changed to IBM Data Server Client v11.1 (fix pack: v11.1.4fp6_linuxx64_server_t.tar.gz from https://www.ibm.com/support/pages/node/6427871 -> https://www.ibm.com/support/fixcentral/swg/downloadFixes?parent=ibm%7EInformation%2BManagement&produ...) and got this error: "SQL_ERROR SqlState: NativeError: -1390".

Looking through the Qlik manual "Qlik Replicate server on Linux", we executed these commands but these steps did not resolve the error.

 [

1. Run the following command:
   /opt/ibm/db2/V10.5/bin /db2cli writecfg add -database <db_name_from_endpoint_connection_settings> -host <server_name_from_endpoint_connection_settings> -port <port_from_endpoint_connection_settings>
2. Restart Qlik Replicate Server.

]

Link we referenced: https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/IBM%20DB2%20for%20LUW/prer...

Logs from the "Test Connection":

00012682: 2022-03-10T12:44:50 [AT_GLOBAL       ]I:  Attunity Replicate Server Log (V2021.11.0.165 ip-10-45-123-104.us.aegon.com Linux 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Mon Jan 17 07:06:06 EST 2022 x86_64 64-bit, Revision:6710baac24c6d5fa0f402198f4ad9353ce6cf7c4, PID: 12681) started at Thu Mar 10 12:44:50 2022  (at_logger.c:2680)

00012682: 2022-03-10T12:44:50 [AT_GLOBAL       ]I:  Licensed to TRANSAMERICA, term license (200 days remaining), sources: (FileChannel,DB2zOS,DB2LUW), targets: (FileChannel,PostgreSQL,Kinesis,S3,ODBC), all hosts  (at_logger.c:2683)

00012682: 2022-03-10T12:44:50 [AT_GLOBAL       ]I:  Log file encryption base "lX56jhm7ERGzG4F5i1auOg=="  (at_logger.c:2695)

00012682: 2022-03-10T12:44:50 [TASK_MANAGER    ]I:  Feature flags for process are:  (ar_feature_flags.c:232)

00012682: 2022-03-10T12:44:50 [SERVER          ]I:  Client session (ID 783121950) allocated  (dispatcher.c:274)

00012701: 2022-03-10T12:44:50 [REST_SERVER     ]W:  Failed to initialize pam configuration, pam authentication disabled.  (atctl_serve.c:275)

00012701: 2022-03-10T12:44:50 [REST_SERVER     ]I:  Agent was started successfully (listening on port 3552).  (atctl_serve.c:402)

00012743: 2022-03-10T12:45:16 [SERVER          ]I:  Going to connect to ODBC connection string: Driver={IBM DB2 ODBC DRIVER};Database=DBxxx;PROTOCOL=TCPIP;HOSTNAME= dbhost1.com;SERVICENAME=xxxxx;UID=dbuidxx;  (odbc_endpoint_imp.c:2899)

00012743: 2022-03-10T12:45:16 [SERVER          ]E:  RetCode: SQL_ERROR  SqlState: IM004 NativeError: 0 Message: [unixODBC][Driver Manager]Driver's SQLAllocHandle on SQL_HANDLE_HENV failed [1022502]  (ar_odbc_conn.c:584)

00012743: 2022-03-10T12:45:16 [SERVER          ]E:  Cannot connect to ODBC provider [1022507]  (odbc_endpoint_imp.c:4730)

00012809: 2022-03-10T12:45:53 [SERVER          ]I:  Additional properties = '(null)'  (db2luw_endpoint_imp.c:557)

00012809: 2022-03-10T12:45:53 [SERVER          ]I:  'Dynamic buffer size' feature is enabled = '0'  (db2luw_endpoint_imp.c:559)

00012809: 2022-03-10T12:45:53 [SERVER          ]I:  DB2LUW connection string: DRIVER={IBM DB2 ODBC DRIVER};HOSTNAME=dbhost2;SERVICENAME=yyyy;DATABASE=DByyyy;Patch2=15;XMLDeclaration=0;UID=userID;PWD=***;  (db2luw_endpoint_imp.c:198)

00012809: 2022-03-10T12:45:53 [SERVER          ]E:  RetCode: SQL_ERROR  SqlState:       NativeError: -1390 Message: [IBM][CLI Driver] SQL10007N Message "0" could not be retrieved.  Reason code: "3". [1022502]  (ar_odbc_conn.c:584)

00012809: 2022-03-10T12:45:53 [SERVER          ]E:  Cannot connect to DB2 LUW Server [1022500]  (db2luw_endpoint_imp.c:567)

00012861: 2022-03-10T12:46:14 [SERVER          ]I:  Going to connect to ODBC connection string: Driver={IBM DB2 ODBC DRIVER};Database=DBxxx;PROTOCOL=TCPIP;HOSTNAME= dbhost1.com;SERVICENAME=xxxx;UID=dbuidxx;  (odbc_endpoint_imp.c:2899)

00012861: 2022-03-10T12:46:14 [SERVER ]E: RetCode: SQL_ERROR SqlState: NativeError: -1390 Message: [IBM][CLI Driver] SQL10007N Message "0" could not be retrieved. Reason code: "3". [1022502] (ar_odbc_conn.c:584)

00012861: 2022-03-10T12:46:14 [SERVER          ]E:  Cannot connect to ODBC provider [1022500]  (odbc_endpoint_imp.c:4730)

[john_wang · Accepted solution]

Hello @parthiTA , copy @Heinvandenheuvel ,

Thanks for your update.

The error changed to 

2022-03-10T12:45:16 [SERVER          ]E:  RetCode: SQL_ERROR  SqlState: IM004 NativeError: 0 Message: [unixODBC][Driver Manager]Driver's SQLAllocHandle on SQL_HANDLE_HENV failed [1022502]  (ar_odbc_conn.c:584)

in general it's an common environment problem eg some settings were not set correctly, or unixODBC not installed correctly etc, "isql" is the best tool to troubleshoot this type errors (see below detailed info).

I'd like to suggest to:

1. check your file "site_arep_login.sh" under <Rep_Install_folder>\bin

    Add below lines (modify it to fit your installation):    

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/attunity/replicate/lib:/opt/ibm/db2/V11.5/lib64
export DB2INSTANCE=db2inst1

2. Restart services , and try again.

3. If the problem persist or a new error pops up, you may use "isql" to test the connectivity between the DB2 Client and DB2 Server: edit the "/etc/odbc.ini", "/etc/odbcinst.ini" then run "isql -v" ("-v" - enable the Verbose mode then rich information will be returned). If the isql works then Replicate should work too.

Feel free to let me know if you need any additional assistance.

Regards,

John.

[parthiTA]

Hi, John,

Those steps really helped (update to /etc/odbc.ini). Now the connectivity to the source & target databases works. the Log stream (CDC) works. However, full load is throwing error 1011400.

Is this related to permission or memory?

  LITIES       ]E:  The State Manager cannot allocate agent for component 'st_0_AWS - Target - DBTARGET' [1021400]  (statemanager.c:1117)

00008383: 2022-03-14T09:33:41 [UTILITIES       ]E:  The State Manager cannot allocate new agent [1021400]  (statemanager.c:405)

00008383: 2022-03-14T09:33:41 [STREAM_COMPONEN ]E:  The Stream Component cannot register at State Manager [1021400]  (streamcomponent.c:1642)

00008383: 2022-03-14T09:33:41 [TASK_MANAGER    ]E:  Failed while preparing stream component 'st_0_AWS - Target - DBTARGET'. [1021400]  (subtask.c:853

[john_wang]

Hello @parthiTA ,

Glad to hear the progress. However it's hard to tell where the problem is, source side, or target side.

If I understood correctly, your target is DB2 LUW too - means you are using generic ODBC target , you may use "isql -v" to troubleshoot as well, and also please choose the "DB2" Syntax in the target endpoint, just like:

if the problem persists, please show more complete lines (obfuscating sensitive information as needed) to help us to understand the issue.

thank you,

John.

[parthiTA]

Hi, John,

yes, both source & target are DB2 LUW.

Attaching a verbose logs.

Regards

Parthiban

[Heinvandenheuvel · Accepted solution]

UPDATE: We have solved and drifted away from the original question seemingly into an unrelated area. For the benefit of other looking for that old or that new problem please consider starting a fresh topic with an appropriate subject should more help be needed.

That said...

My reading of the error " cannot allocate agent" is that it is an Replicate internal structure confusion, not a connectivity issue. The section following "Going to connect to ODBC connection string: Driver={IBM DB2 ODBC DRIVER};Database=DB2LUWTargetDB" suggests that the connection is fine.

Now the task is labelled "DB2LUWSourceDBDB - DB2LUWTargetDB Full Load", but it starts with "running CDC only in resume mode". Now that's fine, you can pick any name you like (no matter how misleading 🙂 ) and I certainly have changed tasks on the fly myself many times and have them no longer live up to their flag, but it still is a 'red flag'. Maybe there are more inconsistent settings.

You might want to create a whole new task and start fresh - and for your own sanity make the name short, descriptive, and without spaces.  Maybe DB2_onprem2cloud[_cdc] ?

Be sure to raise a formal support case or more appropriately Professional Services, if  the suggestions provided here leave you stuck. This being  fresh install, was there no (pre)sales team associated to help you get going and a POC perhaps - where is that info? Did it not work at that time? Was there no Professional Service bundled in the sale?

Good luck,

Hein

Hein

[parthiTA]

Hein,

Thanks for the tips.

We opened a Support Case, and will submit a separate topic if required.

Regards

Parthiban