one-to-many replication

[RichJ]

We have replica tasks from MS SQL server as well as PostgreSQL to Netezza (NZ), and we are migrating NZ to snowflake (SF).  So all replica tasks have two versions: one is to NZ, the other is to SF;  but two versions have the same source db. 

The question is this - can we run both versions of a replica task with single source endpoint simultaneously without affect each other? we need that runs for (say) one week before cut-over from NZ to SF.  

The source endpoint configurations includes: ODBC with CDC for PostgreSQL, ODBC with CDC for MS SQL Server, CDC using online TLOGs, CDC using backup TLOGs.

Thanks for help.

[Michael_Litz]

Hi Rich,

Yes, I believe that using a Logstream task will do just what you want.

Please check out the documentation for this task configuration and let me know what you think.

https://help.qlik.com/en-US/replicate/May2022/Content/Replicate/Main/Log%20Stream%20Staging/intro.ht...

Thanks,
Michael

[Michael_Litz · Accepted solution]

Hi Rich,

Yes, I believe that using a Logstream task will do just what you want.

Please check out the documentation for this task configuration and let me know what you think.

https://help.qlik.com/en-US/replicate/May2022/Content/Replicate/Main/Log%20Stream%20Staging/intro.ht...

Thanks,
Michael

[lyka]

Hi Rich,

@Michael_Litz is correct. You can achieve this by using logstream.

You can refer to this article from our community:

https://community.qlik.com/t5/Knowledge/Log-Stream-Staging-The-Why-and-the-How/ta-p/1712077

Thanks

Lyka

Thanks

Lyka

[Dana_Baldwin]

Hi @RichJ 

If you plan to run 2 different tasks pulling from the same source but writing to different targets, I don't see an immediate concern. There will be some additional overhead for the additional connections to the source, so please test. If by "task with single source endpoint" you mean one task that writes to two different targets, that is not possible. You could use Log Stream, which actually would use 3 tasks: one to stage all the changes, one to write to NZ, and one to write to SF - but avoids additional overhead on the source as there would only be one connection reading from there.

Thanks,

Dana

[RichJ]

I might not be clear.  That's  two different tasks but share the same single source endpoint.  The target endpoints are pointing to two different databases.   Looks like using tlog files will be ok.  But how about single source endpoint (postgreSQL or MS SQL Server) and connection type is "ODBC with CDC"?  

The simple tests are past.  But for longer term with large operations in PROD env,  the firm confirmation is desirable.

Thanks.

[Dana_Baldwin]

Thanks @RichJ 

Support generally isn't involved in setting up & testing of ODBC endpoints, our Professional Services Team does that work - so unfortunately I don't have any knowledge / experience to draw upon. Since your configuration is working fine with only one task using each source endpoint, at this point I would assume that going with a Log Stream configuration would be the safest. You might consider reaching out to the Professional Services person you worked with setting up ODBC if Log Stream is not desirable - I could understand that given that you only need this for a limited amount of time.

Sorry I could not be of more assistance.

Dana

[Barb_Fill21]

Hi @RichJ   

Re: ODBC connection as the source,, If Replicate has a Source Endpoint for a database type, then the ODBC type Source Endpoint is not allowed. 

For example, if you have an Oracle Source database, and want to use the ODBC for CDC to read from the source, it is not allowed as we have an Oracle Source db endpoint for that type of database. 

Just an FYI.   🙂 

Sincerely,

Barb