One popular configuration option for Replicate tasks is to handle a Many-to-One scenario. Imagine regional source sales databases that you want to consolidate into a single target database in order to simplify reporting needs across regions. This article will describe how to set up and configure replicate tasks to handle this situation, and introduce some key concepts for this to work.
Lets start with the source databases.
Key Concept The source database tables structure must be identical across the many source database.
Key Concept No guarantee that the primary key values on tables will be unique across the many source databases.
Each source table primary key will have its own unique key values. If these values are duplicates in other source database tables there will be conflicts in the target database as the many sources get replicated to the target ( There would be duplicate primary key issues).
NOTE: This potential conflict will be taken care of in the task design/transformation
When configuring Replicate tasks to handle a Many to One scenario you will need to have a task for each source database. Each task will do Full Load and CDC to the same target database thus creating the merged database that contains all the tables and values from each source in one target database. In my example I will only have one table in my tasks called customer. My example has (Region_1 to Region_ALL) and (Region_2 to Region_ALL) tasks configured.
For each task you define there will be a unique source endpoint and the identical target endpoint.
There are 2 source endpoints and only one target endpoint defined, as each task will use the same target endpoint.
Key Concept Each task must be set to Do Nothing on the Full Load/Full Load Settings/Target Table Preparation screen. If the tables do not exist then the task will create them. This setting insures that as each task runs its full load the task will not drop or delete records from the previous task full load. NOTE: It is okay to set this during the task creation as the task will still create the tables if they do not already exist.
Key Concept Table level transformation adding a new field to the target side table that becomes part of the tables primary key. You can see the table transformation for the Customer table in this image.
The field RegionCode has been added to the target table Customer, then click on the "Key" column to add it to the PK on the target. For each task hard code a unique number for the region code. i.e. 1, 2, 3 etc.
NOTE: This does not impact the source table, it is target side only.
Lets have a look at the target side database/tables. You can see that the RegionCode field has been added to the table and it is part of the PK, thus insuring there will be no duplicate key errors.
The customer table structure.
The customer table showing the merged records from both Region 1 and Region 2 source databases.
NOTE: Please also watch the accompanying video on how to configure this many to one environment.
The information in this article is provided as-is and to be used at own discretion. Depending on tool(s) used, customization(s), and/or other factors ongoing support on the solution above may not be provided by Qlik Support.