How to process Sql server data (Approx 1.3 millions records) in Talend in chunks

Vthota1687286448 — Fri, 15 Nov 2024 21:29:38 GMT

How to process SQL server data (Approx 1.3 million records) in Talend in chunks

Hi,

We need to process Sql server data [Approx 1.3 millions) in Talend. When we are trying to process all the records in one go, it's giving "Heap space - out of memory" error in Talend. We have tried all the ways to increase the JVM memory size but it's not working out. Probably because we have complex logic in worflow.

So now we are looking to process data in chunks but not sure how can it be done in Talend. Currently, we are pulling sql server data using the "SqlserverInput" component.

I am not using tMap and its direct load, no transformation

Could anyone please advise on this?

Thanks in advance!!

Regards,

Ram

Re: How to process Sql server data (Approx 1.3 millions records) in Talend in chunks

David_Beaty — Mon, 02 Oct 2023 09:40:57 GMT

Without seeing what your job is doing it’s hard to say, giving the Talend job extra memory isn’t really the solution here. For component like TMap, sort and uniq, they have advanced parameters to utilise temporary files on the server as an offline memory story to stop this.

https://help.talend.com/r/en-US/8.0/tmap/tmap?tocId=rNWfV5QzQlTBh66dwlMRMA

like temp data directory path.

topic How to process Sql server data (Approx 1.3 millions records) in Talend in chunks in Talend Studio

How to process Sql server data (Approx 1.3 millions records) in Talend in chunks

Re: How to process Sql server data (Approx 1.3 millions records) in Talend in chunks