topic Re: Loop in folders for files and records in Talend Studio

Loop in folders for files and records

Anonymous — Wed, 18 Apr 2018 05:52:31 GMT

I need a help to find loop handling

Scenario :

I have 5 files with 100 records each.

I want to take 1st file and load 10 records as loop and load into db. Once first file is over then 2nd file and again load 10 records as loop and load into db. It should iterate for all 5 files.

Can anyone help me?

Thanks in Advance

Re: Loop in folders for files and records

TRF — Wed, 18 Apr 2018 07:21:30 GMT

Hi,
Use tFileList to iterate over the file list, then tFileInputDelimited (or other depending on the files type) to get the records.
In this component, use the variable tFileList_1_CURRENT_FILEPATH as the filename and set the field "Limit" to the number of records you want to read.

Re: Loop in folders for files and records

Anonymous — Wed, 18 Apr 2018 07:42:32 GMT

Thanks for your response

If I am setting "Limit" as 10 then its taking only 1st 10 record of each file.

But I want this to read full file means 100 records so 10 times it should iterate then it should go for 2nd file

Re: Loop in folders for files and records

TRF — Wed, 18 Apr 2018 08:39:34 GMT

Sorry, I misread your request.

The easiest way is to split the input files into 10-line files.

The process should look like this:

tFileList-->tFileInputDelimited-->tFileOutputDelimited
| onSubjobOK
tFileList-->tFileInputDelimited-->t<DBName>Output

In 1rst subjob, you iterate over the input files and push the result into small output files.

In the tFileOutputDelimited, tick the option "Split output in several files" and enter 10 into the field "Rows in each output file".

In the 2nd subjob you will iterate over the list of "small" input files, read the content and push it into your database.

Re: Loop in folders for files and records

Anonymous — Wed, 18 Apr 2018 08:49:18 GMT

Thanks TRF

Splitting file is not allowed here. we want to use same file.

Re: Loop in folders for files and records

TRF — Wed, 18 Apr 2018 10:14:18 GMT

Could you explain why?

Re: Loop in folders for files and records

Anonymous — Wed, 18 Apr 2018 10:36:45 GMT

We have given suggestion of splitting file but people saying it will cause i/o operation so need to avoid it.

So only

Re: Loop in folders for files and records

TRF — Wed, 18 Apr 2018 10:52:10 GMT

How many files in real life?

How many records?

What's the job frequency?

Re: Loop in folders for files and records

Anonymous — Wed, 18 Apr 2018 10:57:37 GMT

Records - 25 Million

approximately 10 files

25 Million is one time but 3 Million weekly

Re: Loop in folders for files and records

Anonymous — Wed, 18 Apr 2018 11:00:59 GMT

Records - 25 Million

approximately 10 files

25 Million is one time but 3 Million weekly

I need to process in batch

Can you give some suggestion?

Re: Loop in folders for files and records

TRF — Wed, 18 Apr 2018 11:09:56 GMT

I strongly suggest you to try the suggested approach.

3 million records per week, not a so huge volume, and Talend is very fast with this kind of files, it could be a good solution despite of the opposition regarding the i/o operations.

Else, you will have to play with tFlowToIterate / tLoop / tIterateToFlow to build the right algorithm.

Re: Loop in folders for files and records

Anonymous — Wed, 18 Apr 2018 11:22:08 GMT

Thanks TRF

tFlowToIterate / tLoop / tIterateToFlow

how to use mentioned component and where ?