Unlock a world of possibilities! Login now and discover the exclusive benefits awaiting you.
Hi,
We have Talend job (Loading Fact Table in DW) in which we use more than 4 lookup tables. Among those lookup tables only one table has large amount of data (10 Million). Whenever the Job is executed we get error as "Out of Heap Memory".
RUN 1 :
As suggested in Talend Help Site i have tried to Increase the JVM parameters. Even after increasing the Job am unable to execute the Job.
JVM Parameters:
Xms256M
Xmx1610M
Source - SQL Server
Lookup/ Target - Oracle.
In Each of the lookup table we have enabled CURSOR.
RUN 2:
Also tried to load with lookup data stored in the Local Directory by enabling the Store Temp Data directory in tMap.
The problem with this method is We are unable to Load all the data from source to Target. For Example : If the Source has 10 Millions records am able to Load only Half Million record into Target (Meaning lookup fails for not processed records).
Also the time taken is more to process.
Please Note:
RAM Speed - 4GB
In both these we were unsuccessful, is there any way in talend to Handle the lookup effectively.??
If so please let us know..!! Inputs would be helpful..
Below i have also attached my Job Screen Shot:
We have Talend job (Loading Fact Table in DW) in which we use more than 4 lookup tables. Among those lookup tables only one table has large amount of data (10 Million). Whenever the Job is executed we get error as "Out of Heap Memory".
RUN 1 :
As suggested in Talend Help Site i have tried to Increase the JVM parameters. Even after increasing the Job am unable to execute the Job.
JVM Parameters:
Xms256M
Xmx1610M
Source - SQL Server
Lookup/ Target - Oracle.
In Each of the lookup table we have enabled CURSOR.
RUN 2:
Also tried to load with lookup data stored in the Local Directory by enabling the Store Temp Data directory in tMap.
The problem with this method is We are unable to Load all the data from source to Target. For Example : If the Source has 10 Millions records am able to Load only Half Million record into Target (Meaning lookup fails for not processed records).
Also the time taken is more to process.
Please Note:
RAM Speed - 4GB
In both these we were unsuccessful, is there any way in talend to Handle the lookup effectively.??
If so please let us know..!! Inputs would be helpful..
Below i have also attached my Job Screen Shot:
I didnt try breaking the job, i will give a try on that
could be a better solution
If it's possible join your data in several tmap & release memory between them (a temp stored in file for exemple as I/O in file is the fatest way)
Could be
1. join with big data storing on disk or reload each time - empiric testing
& store result in flat file
2 read result and join other data
ps: solution by increasing jvm heap size is not a perennial solution as if data volume increase , it failed again (in production) .. so it could become a none solution as you cannot increase jvm indefinitely.
Could be a solution for stable volume of data.
hope it help
regards