Hi,
Lets say i have two hdfs files A and B. File A with 40 columns. I am generating 10 million data for file A. While generating data for file B, i need to take random records from file A (like 7 or 8 columns only primary key - foreign key relationship). As i am giving 10 million data as lookup, i am getting GC overhead issue. I have 8GB RAM and i have given XMX till 8096. I tried with 4096 as well.
Please give a solution to solve this issue / an alternate method to take random records.
As i am generating data dynamically, i think breaking the files in to small files and fetching random records is not possible. Please clarify.
I have used a temporary location also for tmap and increased buffer size too. But i couldn't find whether its working. Please let me know if there is a way to check whether temp data is getting stored in this location and there is a effect in increasing buffer size.
Please appreciate our Talend community members by giving Kudos for sharing their time for your query. If your query is answered, please mark the topic as resolved
Thank you for your suggestion. But as i have said already i used temporary location already for storing temp data. Still I get memory issue. XMX size of 8096 is not enough for this lookup but i cannot give higher xmx either.
When i store temp data on disk, i am getting error message in my desktop too (Saying something like- " an application is using lot of space,close the program?")
