topic Limiting an IterateToFlow in Talend Studio

Limiting an IterateToFlow

PhilHibbs — Sat, 16 Nov 2024 03:02:16 GMT

I have a component that produces an iterate link, in this case it's a tS3List but the same could apply to a tFileList etc.

I then have a tIterateToFlow, I do some manipulation, and then a tFlowToIterate.

When there are a few hundred thousand iterates, I get an Out of Memory error.

How can I throttle the iterate so as to prevent this error? Is there any way of limiting the number of iterations, or breaking them up into chunks?

Re: Limiting an IterateToFlow

manodwhb — Tue, 10 Mar 2020 03:06:16 GMT

@PhilHibbs ,could you please share your job design ?

Re: Limiting an IterateToFlow

PhilHibbs — Tue, 10 Mar 2020 08:59:33 GMT

I suspect that my problem may have been caused - or amplified - by a tLogRow set to table mode... it would still be nice to be able to break up large workloads like this though.