Loop in folders for files and records

[Anonymous]

Hi  

I need a help to find loop handling 

Scenario :

I have 5 files with 100 records each.

I want to take 1st file and load 10 records as loop and load into db. Once first file is over then 2nd file and again load 10 records as loop and load into db. It should iterate for all 5 files.

Can anyone help me?

Thanks in Advance 

[TRF]

Hi,
Use tFileList to iterate over the file list, then tFileInputDelimited (or other depending on the files type) to get the records.
In this component, use the variable tFileList_1_CURRENT_FILEPATH as the filename and set the field "Limit" to the number of records you want to read.

[Anonymous]

Thanks for your response 

If I am setting "Limit" as 10  then its taking only 1st 10 record of each file.

But I want this to read full file means 100 records so 10 times it should iterate then it should go for 2nd file

[TRF]

Sorry, I misread your request.

The easiest way is to split the input files into 10-line files.

The process should look like this:

tFileList-->tFileInputDelimited-->tFileOutputDelimited
| onSubjobOK
tFileList-->tFileInputDelimited-->t<DBName>Output

In 1rst subjob, you iterate over the input files and push the result into small output files.

In the tFileOutputDelimited, tick the option "Split output in several files" and enter 10 into the field "Rows in each output file".

In the 2nd subjob you will iterate over the list of "small" input files, read the content and push it into your database.

[Anonymous]

Thanks TRF

Splitting file is not allowed here. we want to use same file.

[TRF]

Could you explain why?

[Anonymous]

We have given suggestion of splitting file but people saying it will cause i/o operation so need to avoid it.

So only 

[TRF]

How many files in real life?

How many records?

What's the job frequency?

[Anonymous]

Records - 25 Million

approximately 10 files

25 Million is one time but 3 Million weekly

[Anonymous]

Records - 25 Million

approximately 10 files

25 Million is one time but 3 Million weekly

I need to process in batch 

Can you give some suggestion?