ETL Consolidating Multiple Databases into 1

Anonymous — Thu, 21 Mar 2019 01:11:13 GMT

I'm not sure if this is the correct location or not, however, I didn't find another that really fit. I'm hoping that I'm able to accomplish a rather large ETL task with talend. I'm not sure how to even get started on this and do plan on looking at tutorials or something as well. I have a customer who has 4 databases several hundred GB without BLOB and 6.5 TB with BLOB. For the most part the schema is identical, however, there are tables in database B that isn't in database A, C or D. Am I able to use talend to accomplish this task? If so, how do I go about doing this without having to go table by table? There's about 400 tables in the databases and that would take a very long time to go through each one. Or, is this something that will have to be done for each table? It's so hard to find any training on ETL.

Re: ETL Consolidating Multiple Databases into 1

vapukov — Thu, 21 Mar 2019 07:38:01 GMT

Hi,

Talend (and really may be any other) - not a tool for automatic sync databases, like "one click and all work" 🙂

but problem mostly not in the structure, this is could be extracted and stored into the repository

the biggest problem - what you plan to do with keys?

each database have its own unique keys, and some tables depend on these keys and can not be just simple append, logic usually much more complicated
plus incremental loading and other pain-points

for compare and sync structure - many good tools dedicated for this task - dbdiff, dbcompare, sbschema etc (depend from database, but possible to find for many most popular)

for real sync - first what to need to check is proper database design and native tools like replication (and or foreign tables/servers)

Talend is part of ETL tools, they not for automatic transfer with automatic resolving issues and collisions, they for design complicated transformations and logic

add:

as usual first request - we want all and immediately! 🙂

but in fact it only a few tables need to have tomorrow, and here talend could be a good tool

topic ETL Consolidating Multiple Databases into 1 in Talend Studio

ETL Consolidating Multiple Databases into 1

Re: ETL Consolidating Multiple Databases into 1