topic Merge Source Databases to AWS Aurora MySQL Many to 1 either replication or ETL in Qlik Replicate

Merge Source Databases to AWS Aurora MySQL Many to 1 either replication or ETL

CloudRocks — Tue, 30 Nov 2021 22:25:56 GMT

Hi,

I am trying to ETL or replicate data from AWS RDS SQl Server to AWS Aurora MySQL. I have 3 source databases with same schema in RDS SQL Server SOurceDB1, SourceDB2, SourceDB3. I would like to ETL/replicate them to single database to Aurora MySQL TargetDB using Qlik .TargetDB has same schema as all 3 sources. Is this possible to ETL or replicate minute by minute to a single Aurora target on daily basis or can they only be replicated to 3 targetDB's ?, Which Qlik tools can be used, what limitations I need to consider other than SQLServer vs MySQL DB differences ?, Also does Qlik has ETL capability , with transformations in between source and target ?

Example:Lets say 3 source DBs have 100 records each i.e 300. The target MySQL Aurora single Database will store all 300 records. Also when a record is added , updated to any of the source DB same should reflect in the target DB

Re: Merge Source Databases to AWS Aurora MySQL Many to 1 either replication or ETL

Dimitri_Remez — Wed, 01 Dec 2021 10:32:27 GMT

Hi Prashant,

Thank you for posting this question! I will try to answer to your points below:

To achieve the goal you described, you may use the Qlik Replicate product and configure a many to one environment.
Many to one replication: there are some key concepts to keep in mind that are covered in the video and in the useful link that I'm listing below:
- Qlik Replicate: How to configure a many-to-one replication: https://www.youtube.com/watch?v=0mka2LNf7Sg
- Qlik Replicate: Many-to-One Replication Configuration: https://community.qlik.com/t5/Knowledge/Qlik-Replicate-Many-to-One-Replication-Configuration/ta-p/1787230
You can also use our Change Data Capture Technology and choose either a Transactional apply or Batch optimized apply mode (minute by minute if you prefer) to capture and replicate transactional changes from the source and move them to the target.
- Change Processing Tuning: https://help.qlik.com/en-US/replicate/November2021/Content/Global_Common/Content/SharedEMReplicate/Customize%20Tasks/tasks_applychangtunestab.htm
The limitations and the prerequisites to consider depend on each endpoint that you use:
- Using Amazon RDS for SQL server as a source: https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/AWS%20RDS%20MSSQL/aws_SQLServer_DB_source.htm
- Using a MySQL-based database as a target: https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/MySQL/MySQL.htm
Yes, Qlik Replicate offers transformation capabilities like the ones that you may find described in the below links:
- Customizing tasks: https://help.qlik.com/en-US/replicate/November2021/Content/Global_Common/Content/SharedEMReplicate/Customize%20Tasks/cust_tasks.htm
- Using the Transform tab: https://help.qlik.com/en-US/replicate/November2021/Content/Global_Common/Content/SharedEMReplicate/Customize%20Tasks/tasks_useTransTable.htm
- Transformation action: https://help.qlik.com/en-US/replicate/November2021/Content/Global_Common/Content/SharedEMReplicate/Customize%20Tasks/tasks_defGlobTransRule.htm
- Please, also have a look at the Qlik Replicate help and User Guide for more information: https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/Introduction/Home.htm
  
  I hope this information helps to answer your questions.

Kind regards,
Dimitri

Re: Merge Source Databases to AWS Aurora MySQL Many to 1 either replication or ETL

CloudRocks — Tue, 07 Dec 2021 16:58:29 GMT

Hi Dimitri,

Thank you for your reply. We would actually like to try the MS SQL RDS Many DBs to 1 MySQL Aurora replicate . When I try to use the Qlik replicate, it is gives me a standard trial of Mysql to postgres for 4 hours. Is there any way we can try our actual use case ?. Then only we can decide if we want to move forward with you . There is one other vendor who let us use his products for a limited time, the way we wanted to at a smaller scale. Also we will need time more than 4 hours as well. If you can accommodate for this that will be good.

Kind Regards,

Prashant(PrimeTSR)

Re: Merge Source Databases to AWS Aurora MySQL Many to 1 either replication or ETL

john_wang — Sat, 11 Dec 2021 07:14:15 GMT

Hello @CloudRocks ,

Yes, you can contact your region Qlik sales representatives, or visit Contact Sales page , the sales team may help you to conduct a PoC , answers our questions, and provide you a trial license so you can try more with Qlik Replicate.

Looks to me Replicate fit all you above doubts/requests. Let me know if you need any additional assistance.

Regards,

John.