Qlik Community

QlikView App Dev

Discussion Board for collaboration related to QlikView App Development.

Announcements
QlikWorld online is next week! REGISTER NOW
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Job Failing on left join with 2 Billion records against 9 million records

Hi,

I am trying to execute a job by joining two tables, one with 2 Billion records against the other with 9 million records. Job is failing to execute, throwing out a general script error. we have around 400GB RAM on our server.

Could some one explain the limitation with joins in Qlikview.

Thanks.

5 Replies
jonathandienst
Partner
Partner

That is a large join. You may simply be running out of RAM. Have you monitored memory use of the model? Is it in an outer join? What is the cardinality of the join key (the common fields)? Are there common fields?

Logic will get you from a to b. Imagination will take you everywhere. - A Einstein
Gysbert_Wassenaar

Is the join necessary in the first place? If you don't need to do calculations in the script that use fields from both tables together you probably don't need to join them. Perhaps using Keep instead of Join will do the job. Or perhaps you can use a mapping table and the Applymap function instead of a join.


talk is cheap, supply exceeds demand
Not applicable
Author

Thank you for the reply Jonathan,

I have 400GB RAM in my server. I hope that is enough for this but I am not sure. And also I didn't monitor memory use of model. I there any way to monitor the memory use in qlikview itself or do I have to check it in Task manager?

Also is there any way to calculate the amount of RAM that requires for execution of the file?

Not applicable
Author

Thank you for the reply Gysbert. I have few calculations in the table such as formatting the dates, setting up the flags and concatenating the fields.

May I know how fast the execution gonna be, if I go for keep instead of join?

Gysbert_Wassenaar

If you don't need to do calculations that involve two fields from two different tables then you probably don't need a join to make sure the two fields are in a single table. A keep will be a lot faster since it won't have to create a new table.


talk is cheap, supply exceeds demand