Dynamically extract/load multiple tables using metadata?

[MichelLalancette]

We're just about to buy Talend (Studio/Forge not sure on the nomenclature yet), moving from SSIS with the express edition of BIML.

Currently, the way we fetch our data is through the use of metadata saved in SQL and then this is used by BIML to generate the different SSIS packages to load 10 tables at once.

The metadata table contains things such as the connection string, source db, schema, table, WHERE conditions that should be appended as well as the package name to be generated.

All the column metadata is also saved in another table for SQL to generate the Query.

This data then gets picked up by BIML and through loops, it would create Sequence containers with up to 10 table Extracts running in parallel (to not overwhelm the target server).

If the data is loaded in delta, it performs extra delta steps such as moving the data to the archive after landing it. Then, whenever we need to add a new table we update the metadata, rerun the biml script and our dtsx gets rebuilt and we redeploy (can't be automated with the free version).

Is there anything that resembles what we're doing in Talend?

[Dave_Simo]

Hello,

On talend there is the notion of dynamic diagram which allows you to achieve this, below is an example via this link which is close to your needs

https://www.linkedin.com/pulse/dynamic-schema-talend-jean-francois-monteil-piose/

Best Regards