Merge Source Databases to AWS Aurora MySQL Many to 1 either replication or ETL

[CloudRocks]

Hi,

I am trying to ETL or replicate data from AWS RDS SQl Server to AWS Aurora MySQL. I have 3  source databases with same schema in RDS SQL Server SOurceDB1, SourceDB2, SourceDB3. I would like to ETL/replicate them to single database to Aurora MySQL TargetDB using Qlik .TargetDB has same schema as all 3 sources. Is this possible to ETL or replicate minute by minute to a single Aurora target on daily basis or can they only be replicated to 3 targetDB's ?, Which Qlik tools can be used, what limitations I need to consider other than SQLServer vs MySQL DB differences ?, Also does Qlik has ETL capability , with transformations in between source and target ? 

Example:Lets say 3 source DBs have 100 records each i.e 300. The target MySQL Aurora single Database will store all 300 records. Also when a record is added , updated to any of the source DB same should reflect in the target DB 

[Dimitri_Remez]

Hi Prashant,

Thank you for posting this question! I will try to answer to your points below:

1. To achieve the goal you described, you may use the Qlik Replicate product and configure a many to one environment.
2. Many to one replication: there are some key concepts to keep in mind that are covered in the video and in the useful link that I'm listing below:
   • Qlik Replicate: How to configure a many-to-one replication: https://www.youtube.com/watch?v=0mka2LNf7Sg
   • Qlik Replicate: Many-to-One Replication Configuration: https://community.qlik.com/t5/Knowledge/Qlik-Replicate-Many-to-One-Replication-Configuration/ta-p/17...
3. You can also use our Change Data Capture Technology and choose either a Transactional apply or Batch optimized apply mode (minute by minute if you prefer) to capture and replicate transactional changes from the source and move them to the target.
   • Change Processing Tuning: https://help.qlik.com/en-US/replicate/November2021/Content/Global_Common/Content/SharedEMReplicate/C...
4. The limitations and the prerequisites to consider depend on each endpoint that you use:
   • Using Amazon RDS for SQL server as a source: https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/AWS%20RDS%20MSSQL/aws_SQLS...
   • Using a MySQL-based database as a target: https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/MySQL/MySQL.htm
5. Yes, Qlik Replicate offers transformation capabilities like the ones that you may find described in the below links:
   • Customizing tasks: https://help.qlik.com/en-US/replicate/November2021/Content/Global_Common/Content/SharedEMReplicate/C... 
   • Using the Transform tab: https://help.qlik.com/en-US/replicate/November2021/Content/Global_Common/Content/SharedEMReplicate/C...
   • Transformation action: https://help.qlik.com/en-US/replicate/November2021/Content/Global_Common/Content/SharedEMReplicate/C...
   • Please, also have a look at the Qlik Replicate help and User Guide for more information: https://help.qlik.com/en-US/replicate/November2021/Content/Replicate/Main/Introduction/Home.htm
     
     I hope this information helps to answer your questions.

Kind regards,
Dimitri

[CloudRocks]

Hi Dimitri,

Thank you for your reply. We would actually like to try the  MS SQL RDS Many DBs to 1 MySQL Aurora replicate . When I try to use the Qlik replicate, it is gives me a standard trial of Mysql to postgres for 4 hours. Is there any way we can try our actual use case ?. Then only we can decide if we want to move forward with you . There is one other vendor who let us use his products for a limited time,  the way we wanted to at a smaller scale. Also we will need time more than 4 hours as well. If you can accommodate for this that will be good.

Kind Regards,

Prashant(PrimeTSR)

[john_wang]

Hello @CloudRocks ,

Yes, you can contact your region Qlik sales representatives, or visit Contact Sales page , the sales team may help you to conduct a PoC , answers our questions, and provide you a trial license so you can try more with Qlik Replicate.

Looks to me Replicate fit all you above doubts/requests. Let me know if you need any additional assistance.

Regards,

John.