How can I merge master data from different sources and then replicate back to my subscribers
Hi,
I have different applications/sources of data (running back-end on SQL Server, DB2, Oracle, MySQL, etc.). They all have similar customer/product tables. I want to import master tables (customer/product etc.) data from those data sources in one place and based on their Unique Identification (e.g. my unique customer identification number), I manage and update their related data in master database and then sync/export the same to those sources back (replicate). All subscribers then have all customers/product data. If one customer was added from SQL Server, it could be found in Oracle/DB2 database as well and vice versa.
1- I have no idea which tool can help on this?
2- I also want this tool can automatically do data merging activity, replicate back data to subscribers and show the log/dashboard where we can see the flow of data stats which data was inserted/update and what was its source/destination.
3- Minimum user interaction, do not need a person to verify data or do work manually.
Please provide some guidance, how close someone has done such kind of work?
You can use it to merge and output data from multiple sources. One source is your main data input and one or more sources are your "lookups" which are joined in like in SQL. You then define one or more output flows based on expression rules you define.
I need a document and running sample (demo) which I can test and then confirm it works for my solution. Kindly provide any video, documentation for well understanding.
What you are looking for is the whole Talend tool. The entire Talend data integration product is used to build jobs that do what you describe in the OP. You can search the web for learning resources on Talend. tMap is going to be a critical component to use.
Talend has got lot of components in its Palette to do data processing. You can use tMap component if the merging and joining is straightforward type (scenarios are present in tMap sample scenarios in help.talendcom).
But if you need to use advanced matching algorithms, that also is possible in Talend. For example, you can use component like tMatchGroup which can be used to find survivor record from a group of records (sample scenario in below link).
Talend also has lot of native components which can extract and load data at a faster pace. Please try these components from Talend Palette and it should resolve your queries.
Warm Regards, Nikhil Thampi
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved 🙂
