hi everyone , I am new to talend and this might be a repeated question but i couldn't find a suitable answer so asking again.
i have created a job that retrieve data from a table and then it regroups the data using taggreagte component, next i insert the aggregated data into another table'F58SICRA' , the table'F58SICRA' can have both inserts and updates. so i have created 2 temporary tables one for insert 'F58TMPI' (lookup reject) and the other for update 'F58TMPU'(the existing records) to avoid the problem of "duplicated key value", the problem is that when i execute the job it always ends with an error 'out of memory exception' in the lookup process(the lookup table has large volume of data) even after i increased the memory space.
Lookups with a lot of data can cause memory issues if they are left in their default configuration. However, there is an option to "Reload at each row" which can help, particularly if your main record set is significantly smaller than your lookup set. This is explained here....
Another approach is to carry out the lookup in the database. This isn't always possible if the source and the lookup are in different dbs that are not linked. But where the source and lookup data is able to be queried inside the one db, it is often more efficient to join the data in the SQL component used to bring in the data.
Lookups with a lot of data can cause memory issues if they are left in their default configuration. However, there is an option to "Reload at each row" which can help, particularly if your main record set is significantly smaller than your lookup set. This is explained here....
Another approach is to carry out the lookup in the database. This isn't always possible if the source and the lookup are in different dbs that are not linked. But where the source and lookup data is able to be queried inside the one db, it is often more efficient to join the data in the SQL component used to bring in the data.
