topic Re: Data migrate from MongoDb to Sql server in Qlik Replicate

Data migrate from MongoDb to Sql server

ZuhaibRana — Tue, 14 Jun 2022 10:05:38 GMT

i am trying to migrate data from MongoDb to Sql server. MongoDb using as a Source and Sql server using as a target in qlik replicate. i am getting following errors.

Task 'Mongo db to sql server' encountered a recoverable error

Stream component 'st_0_Mongodb' terminated
Stream component failed at subtask 0, component st_0_Mongodb
Error executing source loop

Capture loop failed
Failed to start MongoDB watcher.

An error occurred during capture changes

Failed to start Change Processing from specified time (Timestamp{value=7108998699761532928, seconds=1655192743, inc=0})

Re: Data migrate from MongoDb to Sql server

john_wang — Tue, 14 Jun 2022 12:02:36 GMT

Hello @ZuhaibRana ,

From the error messages looks like your MongoDB source Server is not ready for CDC. Please refer to Qlik Doc, the MongoDB Server must be set up in a replica set configuration . MongoDB does not support watch() function if it's a single node config.

BTW, could you please let me know what's the MongoDB server version? is it an on-premise or Atlas.

Hope this helps.

John.

Re: Data migrate from MongoDb to Sql server

ZuhaibRana — Wed, 15 Jun 2022 05:27:26 GMT

Mongo Db version: 4.0.28

you are saying this is CDC issue. But i am using without CDC.

Re: Data migrate from MongoDb to Sql server

john_wang — Wed, 15 Jun 2022 05:32:32 GMT

Hello @ZuhaibRana ,

Thanks for your update.

The error message is relevant to CDC. if you are intending to run Full Load only then please turn off both "Apply Changes" and "Store Changes", for example the task setting:

Hope this helps.

Regards,

John.

Re: Data migrate from MongoDb to Sql server

ZuhaibRana — Wed, 15 Jun 2022 05:49:42 GMT

Problem Solved Thanks

Re: Data migrate from MongoDb to Sql server

john_wang — Wed, 15 Jun 2022 06:33:19 GMT

Hello @ZuhaibRana ,

Glad to hear that. can you click "Accept as Solution" , it's helpful for others as well.

thanks,

John.

Re: Data migrate from MongoDb to Sql server

ZuhaibRana — Wed, 15 Jun 2022 06:36:38 GMT

now again getting same errors.

Re: Data migrate from MongoDb to Sql server

john_wang — Wed, 15 Jun 2022 08:01:46 GMT

Hello @ZuhaibRana ,

You ever mentioned 5 errors. which error you are facing after you turn off the CDC?

Can you share a piece of task log file (obfuscating sensitive information as needed)? If you are not sure which information is sensitive, I'd like to suggest you open a support case and attach the task log file, our support team will help.

Regards,

John.

Re: Data migrate from MongoDb to Sql server

ZuhaibRana — Wed, 15 Jun 2022 09:33:41 GMT

@john_wang I have solved the problem.

Thanks

Re: Data migrate from MongoDb to Sql server

Sehwag — Sat, 18 May 2024 07:19:06 GMT

Hey ,can you please share your solutions with all steps from scratch .
I shall be greatful ...

Task 'Mongo db to sql server' encountered a recoverable error

Stream component 'st_0_Mongodb' terminated
Stream component failed at subtask 0, component st_0_Mongodb
Error executing source loop

Capture loop failed
Failed to start MongoDB watcher.

An error occurred during capture changes

Failed to start Change Processing from specified time (Timestamp{value=7108998699761532928, seconds=1655192743, inc=0})

Re: Data migrate from MongoDb to Sql server

Vaibhav_19 — Tue, 30 Jul 2024 11:13:13 GMT

Hi @John wang

please help us with

we are Replicating data from MongoDB to BigQuery while prforming CDC we are getting same error as above but we can perform Full-Load easily.

Can you please confirm wheather we can perform CDC on Standlone MongoDB instance.

Following are the error:

1 Failed to start Change Processing from specified time (Timestamp{value=7397330334062215168, seconds=1722325183, inc=0})

2 An error occurred during capture changes

3 Capture loop failed
Failed to start MongoDB watcher.

4 Task 'Mongo_to_BQ' encountered a recoverable error

5 Stream component 'st_0_MongoDB_SRC' terminated
Stream component failed at subtask 0, component st_0_MongoDB_SRC
Error executing source loop

Thanks

Re: Data migrate from MongoDb to Sql server

sureshkumar — Tue, 30 Jul 2024 11:21:15 GMT

Hello @Vaibhav_19

This Error normally happen when your source Monog DB is not compatible with Qlik replicate version.

Kindly let us know what the Qlik Replication version is and Mongo DB version?

Regards,

Suresh

Re: Data migrate from MongoDb to Sql server

john_wang — Tue, 30 Jul 2024 11:35:18 GMT

Hello @Vaibhav_19 ,

A standalone MongoDB instance does not support replication and high availability, hence it's impossible to run MongoDB watcher() API as No OpLog is available in the instance. The minimize requirement is a 3 nodes replica set cluster.

Regards,

John.

Re: Data migrate from MongoDb to Sql server

Vaibhav_19 — Wed, 31 Jul 2024 05:09:05 GMT

Thanks!! SureshKumar and John for the quick response.

@sureshkumar , Using Qlik Replicate Version - MAY-2024 and MongoDB Version - "7.0.7"

Re: Data migrate from MongoDb to Sql server

john_wang — Wed, 31 Jul 2024 06:20:40 GMT

Hello @Vaibhav_19 ,

I'm sure MongoDB version 7 is supported, see Support Matrix of Qlik Replicate 2024.5. The problem is irrelevant to MongoDB version.

Instead, the requirement is that: To enable Replicate to detect changes, MongoDB databases servers must be set up in a replica set configuration.

Hope this helps.

John.

Re: Data migrate from MongoDb to Sql server

Vaibhav_19 — Wed, 31 Jul 2024 06:32:39 GMT

Yes , John you are right the issue is not with MongoDB version.

The Replica set configuration is not enable on MongoDB Database Server we requested for oplog generation on MongoDB Database Server.

Re: Data migrate from MongoDb to Sql server

john_wang — Wed, 31 Jul 2024 06:49:04 GMT

Thank you for your feedback @Vaibhav_19 .

If you do not mind please press the "Accept as Solution" to helps users find answers.

Best Regards,

John.