Does anyone know if the succeeding load technique in loadscript can sometimes use up more memory? I've got a loadscript where part of it looks like this:
Left join load field1, field2, ...field20;
SQL Select field1, field2, ...field20
from bigtable where clause;
It's working with a lot of data - several million rows and it's using up a lot of memory during loading. I don't know whether the memory consumption is driven primarily by the left-join-load operation or the succeeding load technique (which may be storing all of the data from the sql query before doing the load operation).
Re: Succeeding Load Technique -- using extra memory?
The preceding load won't add much overhead afaik (see this blog post and comments). What does happen is that the sql select has to fetch all the records from bigtable that match the where clause. Only after all the records have been retrieved can the left join eliminate the records that don't have a matching record in the table on the left side of the join. If you were to do the left join in the sql select than you'd retrieve only the records that are the result of the join. That means that executing the join on the database server can result in better performance like less memory consumption during a reload.