topic Re: Downloading files and uploading into tables in Talend Studio

Downloading files and uploading into tables

imholzj — Wed, 23 Jan 2019 17:10:38 GMT

I have 15 files that need to be downloaded and then loaded into database tables. The file names don't match the table names.

It seems like I should create a hash so I used tJava to:

java.util.Map<String, String> files2tables = new java.util.HashMap<String, String>();

files2tables.put("file1", "Table1");
files2tables.put("file2", "Table2");
files2tables.put("file3", "Table3");

Put this into globalMap:

globalMap.put("files2tables", files2tables);

Then an array for just the filenames (keys):

globalMap.put("files", files2tables.keySet());

Then I wanted to tForeach, but I can't seem to get the list of files to process into the tForeach.

Is there a better way to approach this problem?

jji

Re: Downloading files and uploading into tables

David_Beaty — Wed, 23 Jan 2019 17:36:12 GMT

Hi,

You can create the list of files using a tFixedFlow component (if they really are fixed file names to table names), and then connect in a tFlowToIterate.

This will loop through each file/table name pair and allow you to read it in and load it into a table. You havent mentioned whether they are the same schema, so I've assumed they are.

Re: Downloading files and uploading into tables

brama — Wed, 23 Jan 2019 17:41:17 GMT

Hi,
My way of doing these is creating a static lookup table in db with filenames and equivalent table names. Take the count of files and use it for the loop. Then in each iteration read the file name from table and its equivalent table name store it in context, then use this context values to insert the data from file to table.
Thanks,
Ramachandran B

Re: Downloading files and uploading into tables

imholzj — Thu, 24 Jan 2019 19:39:52 GMT

Thanks, that's working.

If I want to wait until I download all the files before I continue, what do I test for? Does the tFlowToIterate set a global var when done?

jji

Re: Downloading files and uploading into tables

David_Beaty — Fri, 25 Jan 2019 10:16:45 GMT

I think you'll probably need to upload a screenshot of your job so far for anyone to advise further.

Re: Downloading files and uploading into tables

imholzj — Fri, 25 Jan 2019 14:53:19 GMT

I'd like to download all my files before I move on to loading them into the DB.

Is there a way to complete this IF condition with the size of the tFixedFlowInput_1 Array?

Or is there another "Talend way" of doing this?

jji