topic Re: Iterating through files in Talend Studio

Iterating through files

Anonymous — Sat, 16 Nov 2024 12:19:23 GMT

I have a job right now that loads three CSV files into a DB and then run a series of jobs against that to build out a data warehouse. In the future there may be as many as 5 or 10.
Right now I have a separate job running for each CSV file. What I would like is a single job that goes through a folder running on each CSV that it finds. Does someone have a sample on this?
Thanks,
Luke

Re: Iterating through files

Anonymous — Tue, 27 Mar 2012 19:37:50 GMT

The component you want is tFileList. In that component you can specify a file mask and hook the iterate output to the rest of your processing.

Re: Iterating through files

Anonymous — Sun, 08 Apr 2012 20:33:29 GMT

Hi, does anyone know if there is a way I can pass the output from tFileList into a tRunJob component?

Re: Iterating through files

Anonymous — Sun, 08 Apr 2012 20:53:02 GMT

Ah, looks like you can add a context variable in the job, then set it as a parameter to the tRunJob and then pass this through an iteration between the tFileList and tRunJob:
((String)globalMap.get("tFileList_1_CURRENT_FILEPATH"))

Re: Iterating through files

Anonymous — Fri, 04 May 2012 10:47:54 GMT

I've set up a cron job that pulls AppStore Sales reports to a folder,
which I then let a reoccurring Talend job:
looking in, pulling out files, extracting, loading into a staging table, moves the used files.

-- Update --
Hmm, can't see the files I uploaded...