topic Re: Load data to a table from different format based files located at same location in Talend Studio

Load data to a table from different format based files located at same location

Anonymous — Thu, 06 Dec 2018 05:38:02 GMT

Hi All,

I have one question here.

Input File formats :

.xlsx

.xml

.csv

Output :

Single data base table.

my idea is that we have to convert all input files into one format(example .csv ) and finally load the data into target.

Please suggest how can we achieve this.

Thanks,

Sakri

Re: Load data to a table from different format based files located at same location

ankit7359 — Thu, 06 Dec 2018 05:54:24 GMT

Hi @Sakri,

You can achieve this by doing the following:

Considering different formats(FILE) as main sources, you can use tmap where in you can join and have a single output(tfiledelimited) and then use this output as input for loading data into your database..

As per your definition the flow would go like this:

FILE(INPUT_FILE)-------->

FILE(INPUT_FILE)--------> TMAP ----------->>> TFILEOUTPUTDELIMITED----->>>TDBOUTPUT

FILE(INPUT_FILE)-------->

But there is an advantage and disadvantage in the above scenario...

the advantage is that your performance would be better in the sense FILE TO DB the loading of records is faster. While the disadvantage(as per my view) is that this flow is quite long processes.

You can also try the below FLOW if you wish

FILE(INPUT_FILE)-------->

FILE(INPUT_FILE)--------> TMAP ----------->>>----->>>TDBOUTPUT

FILE(INPUT_FILE)-------->

If you are still unclear on this,Pls get back to community.

Thanks,

Ankit.

Re: Load data to a table from different format based files located at same location

Anonymous — Thu, 06 Dec 2018 06:04:22 GMT

Thank you for quick response @ankit7359

I understood your response and will try this one.

however, i would like to add one more point here that the source files may change dynamically.

example : We have 3 files Today but i may get different format files and number of file may increase or decrease tomorrow based on customer requirement.

Thanks,

Sakri

Re: Load data to a table from different format based files located at same location

ankit7359 — Thu, 06 Dec 2018 06:19:33 GMT

Hi @Sakri,

Do you mean to say that the source files are stored in some remote location or are they stored/sent to you via mail or etc...

Till where i know if they are stored in remote location ---> you have to Internet Components(FTP COMPONENTS) to dynamically bring in your file/files.

Well,if we consider that the source files are coming from REMOTE SEVER/CONNECTION then FTP COMPONENTS have to be used and your flow would be like this...

TFTPCONNECTION------>>>>TFTPFILELIST------->>>>TFILEGET-------->>>>TMAP-------->>>>TFILEOUTPUTDELIMITED--------->>>TDBOUTPUT

TFTPCONNECTION------>>>>TFTPFILELIST------->>>>TFILEGET-------->>>>TMAP--------->>>TDBOUTPUT..

Pls be advised that in TFTPFILELIST,you must specify in the filemask the different source FILE EXT have to be specified.. only then you can work this out...

However, Pls try it from your end and if you are facing issues Pls get back to Community.

Thanks,

Ankit