Big files (tFileInputPositional)

[Anonymous]

Dear Talend Support Team,
We have a huge input file with more than 4 mio. rows in it. This file is read by tFileInputPositional and afterwards its data flow is linked
to tMap. There are in addition lookups with database tables but theses tables don't contain many rows. The problem is the
enormous memory consumption. We need a way to keep the memory moderately. Is there a way to read the huge input file in parts and
than process it and after all read the rest?

Kind regards,
Hilderich

[rbaldwin]

tSortRow can sort on disk under advanced settings. You can then use the tAggregateSortedRow and the list function to denormalize it and reduce the memory consumption.

[Anonymous]

Hello rbaldwin,
That sounds good. I am going to try it tomorrow and give you feedback right here. I am going home now.
Kind regards,
Hilderich

[Anonymous]

Hi hilderich,

Is there any feedback for your issue?
Best regards
Sabrina

[Anonymous]

This approach was helpful and it is in use.