How to process Sql server data (Approx 1.3 millions records) in Talend in chunks

[Vthota1687286448]

How to process SQL server data (Approx 1.3 million records) in Talend in chunks

Hi,

We need to process Sql server data [Approx 1.3 millions) in Talend. When we are trying to process all the records in one go, it's giving "Heap space - out of memory" error in Talend. We have tried all the ways to increase the JVM memory size but it's not working out. Probably because we have complex logic in worflow.

So now we are looking to process data in chunks but not sure how can it be done in Talend. Currently, we are pulling sql server data using the "SqlserverInput" component.

I am not using tMap and its direct load, no transformation

Could anyone please advise on this? 

Thanks in advance!!

Regards,

Ram

[David_Beaty]

Without seeing what your job is doing it’s hard to say, giving the Talend job extra memory isn’t really the solution here. For component like TMap, sort and uniq, they have advanced parameters to utilise temporary files on the server as an offline memory story to stop this.

https://help.talend.com/r/en-US/8.0/tmap/tmap?tocId=rNWfV5QzQlTBh66dwlMRMA

like temp data directory path.