topic Re: Process based on source count data in Talend Studio

Process based on source count data

Anonymous — Wed, 11 Sep 2019 16:05:55 GMT

Hi,

We are using Talend Data Integration 7.x. I have an issue with one of my requirement.

My pipeline consists of components tDBInput -> tMap -> tDBOutput. I have a huge data coming in from source tDBInput(around 10 million) , so I had to filter the data for example (1 to 1,000,000 & 1,000,001 to 2,000,000).

Can I process each filter sequentially such that it will not fail.

I tried to use row-->Iterate: unable to use row(iterate) from tDBInput to tMap.

Please suggest me on the better approach, I cant run in parallel as the data is huge and it might affect the server capacity.

Re: Process based on source count data

Anonymous — Wed, 11 Sep 2019 16:06:53 GMT

@xdshi any suggestions ??

Re: Process based on source count data

TRF — Wed, 11 Sep 2019 16:29:59 GMT

You may have tDBInput -> tFileOutputDelimited (with the number of lines for each files definied on tFileOutputDelimited Advanced settings).
This will create from 1 to n files, then you can iterate over these files and do what you want.

Re: Process based on source count data

TRF — Thu, 12 Sep 2019 09:21:03 GMT

@srkalakonda, does this help?

If so, thank's to mark your case as solved.

Re: Process based on source count data

Anonymous — Tue, 17 Sep 2019 22:39:38 GMT

@TRF, We cannot use files as a intermediate area. Can we iterate based on parameter values ?