Skip to main content
Woohoo! Qlik Community has won “Best in Class Community” in the 2024 Khoros Kudos awards!
Announcements
Nov. 20th, Qlik Insider - Lakehouses: Driving the Future of Data & AI - PICK A SESSION
cancel
Showing results for 
Search instead for 
Did you mean: 
Not applicable

Succeeding Load Technique -- using extra memory?

Does anyone know if the succeeding load technique in loadscript can sometimes use up more memory?  I've got a loadscript where part of it looks like this:

    Left join load field1, field2, ...field20;

    SQL Select field1, field2, ...field20

    from bigtable where clause;

It's working with a lot of data - several million rows and it's using up a lot of memory during loading. I don't know whether the memory consumption is driven primarily by the left-join-load operation or the succeeding load technique (which may be storing all of the data from the sql query before doing the load operation).

2 Replies
Gysbert_Wassenaar

The preceding load won't add much overhead afaik (see this blog post and comments). What does happen is that the sql select has to fetch all the records from bigtable that match the where clause. Only after all the records have been retrieved can the left join eliminate the records that don't have a matching record in the table on the left side of the join. If you were to do the left join in the sql select than you'd retrieve only the records that are the result of the join. That means that executing the join on the database server can result in better performance like less memory consumption during a reload.


talk is cheap, supply exceeds demand
Not applicable
Author

I tracked down my memory issue to a different cause. It had nothing to do with the succeeding load.

Thanks to anyone who thought about the issue.