topic merge multiple databases into one in Talend Studio

merge multiple databases into one

Anonymous — Sat, 16 Nov 2024 08:37:29 GMT

Hello,

I worked previously on merging two mysal and oracle databases into one mysql, here is my basic schema that I used to do the job:

myOracleDbInput --------\ tUnit_1 ----> mySqlDbOutput

mySqlDbDbInput ----------/

Now I'm wondering if I can merge several databases whose parameters are on text files for example located in a directory, for example:
       - Directory: Parameters of the databases:
                             - database1.txt
                             - database2.txt
                              ..

- databaseN.txt
Is there an opportunity to iterate over the directory to achieve this Job?

Thank you.

Re: merge multiple databases into one

TRF — Thu, 01 Mar 2018 15:13:08 GMT

To keep your work simple you may have a main job using a tFileList to iterate over the file list and set the necessary context variables then, call a child job (transmit the whole context) to make the elementary task (for the current context).

Re: merge multiple databases into one

Anonymous — Thu, 01 Mar 2018 15:19:07 GMT

Thank you for your return, Can you explain me more about this method ? what do you mean by a child job !!!?

Re: merge multiple databases into one

TRF — Thu, 01 Mar 2018 15:26:00 GMT

A child is a job you start from another one using a tRunJob component.
This way you're able to keep the main task simple (no iteration) and delegate to the main (or parent) job the task to iterate over file list and set the context variables.
This is a very common design pattern.

Re: merge multiple databases into one

Anonymous — Thu, 01 Mar 2018 15:44:57 GMT

Can you provide an example please ?

Re: merge multiple databases into one

TRF — Thu, 01 Mar 2018 16:01:05 GMT

Currently on my mobile, you'll to wait for hours. However, you can get inspired by other topics with similare question.
Also, if you expect for someone to solve your case for you, you need to give the whole information. For example, what's in your text files? How are they structured?
Even if some people work for free, no one is a magician.

Re: merge multiple databases into one

Anonymous — Thu, 01 Mar 2018 16:14:56 GMT

I'm seeing the other posts, to see if I can do the work, and thank you for your help,

my repertory named "databasesParameters " structured as follows:
database1.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx
            - table_name, xxxx

database2.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx

- table_name, xxxx

databaseN.txt:
            - host, xxxx
            - port, xxxx
            - data_base_name, xxxx
            - user, xxxx
            - password, xxxx

- table_name, xxxx

Real Exemple :

productsDb.txt

            - host,127.0.0.1
            - port, 5487
            - data_base_name, productsDb
            - user, free
            - password, pass

- table_name, construction_products_holding

Thanks again for your help.